Patent Description:
The form factor of a computer input device can have a substantial impact on the user experience with a computing platform. Accordingly, to enhance the user experience, computer input devices have been developed that have a pen or stylus form factor. These devices allow the user to hold the input device in a comfortable position with which the user is familiar (i.e. a comfortable writing position). Further, devices with a pen/stylus form factor support relatively precise control by the user, thereby improving the user experience with a variety of applications, such as drawing and painting applications, writing applications, and the like. However, conventional input devices with a pen/stylus form factor typically have limited positional tracking in two dimensions, thereby limiting utility of the devices for applications that employ three-dimensional spaces, such as virtual reality (VR) and augmented reality (AR) applications. Further, conventional pen/stylus input devices sometimes require the user to write on a special surface, such as a specially-designed tablet or paper, in order to track position of the input device. These special surfaces limit the portability and flexibility of the input device.

Document <CIT> discloses a system for determining the pose of a pen on a surface using infrared sensors, an ultrasonic sensor as well as acceleration detectors and a magnetometer.

<FIG> illustrate techniques for employing an electromagnetic (EM) pose tracking system with a computer input device having a pen or stylus form factor. In some embodiments, a base station device includes one of the transmitter (Tx) or receiver (Rx) module for the EM pose tracking system while the computer input device includes the other of the TX and receiver modules. The EM pose tracking system employs the Tx and Rx modules to collect EM pose data indicating a relative pose between the Tx and Rx modules. Based on the EM pose data, the EM pose tracking system (or a computer device working with the EM pose tracking system) identifies a pose (position, orientation, or both position and orientation) of the computer input device. In some embodiments, the pose is identified in a six degree of freedom (6DoF) space, thereby supporting user input for applications employing such a space, such as a VR or AR application. Furthermore, the EM pose tracking system supports identification of the pose without requiring line-of-sight between the base station and the input device, and without requiring a special writing surface, thereby enhancing the flexibility of the input device. In addition, the components of the EM pose tracking system, such as an EM coil, can be inserted into the input device without requiring a substantial increase in the size of the input device, allowing the input device to maintain the pen or stylus form factor, and thus enhancing the user experience.

For convenience, the term "writing device" is used herein to refer to a computer input device having a pen or stylus form factor as described further herein. It will be appreciated that in some embodiments the writing device can include a writing material, such as an ink reservoir or graphite, that supports writing on a medium (e.g., paper) independent of pose tracking. In other embodiments, the writing device does not include a writing material, but includes a nub or tip that supports use of the device on a display screen or other surface. Further, it will be appreciated that the writing device can be employed for uses other than handwriting, including drawing, selection and manipulation of objects in a computer application, including CAD, AR, or VR application, or any other use for which a stylus can provide input to a computer.

<FIG> illustrates a tracking system <NUM> that employs EM poses to identify poses of a writing device, illustrated as stylus <NUM>. In the depicted example, the tracking system <NUM> includes a base device <NUM> and the stylus <NUM>. The base device <NUM> can be part of or connected to a computer device, a base station independent of a computer device, or can be a mobile device in some embodiments. As described further herein, the tracking system <NUM> is generally configured to identify poses of the stylus <NUM> based on a weighted combination of EM poses (poses generated based on an EM field <NUM> as described further herein) and secondary poses (poses generated based on data provided by a secondary sensor <NUM>). The tracking system <NUM> can therefore be incorporated into a variety of devices and systems that employ pose identification features. For example, in some embodiments the tracking system <NUM> is incorporated in a virtual reality (VR) system to identify the pose of the stylus <NUM> to allow a user to employ the stylus <NUM> to manipulate objects in a VR environment. Thus, in some configurations, the base device <NUM> is a head mounted display (HMD) that displays the VR environment to the user. In other configurations, the base device <NUM> is a device separate from the HMD (such as an accessory or a base-station). It will be appreciated that other configurations of the tracking system <NUM> can be employed. For example, in some embodiments the tracking system <NUM> does not employ a secondary sensor <NUM> and generates poses for the stylus <NUM> based only on EM pose data.

To generate an EM pose, the tracking system <NUM> generates the EM field <NUM>, measures a magnitude and/or phase of the generated EM field <NUM> (referred to herein generally as "EM data"), and computes a relative pose based on the corresponding EM data. In the illustrated embodiment, the EM field <NUM> is generated by the base device <NUM> and the EM data is read at the stylus <NUM>. It will be appreciated that other configurations are possible, including the EM field <NUM> being generated by the stylus <NUM> and the EM data being read at the base device <NUM>, as illustrated below at <FIG>. To support generation of the EM field, the base device <NUM> includes a Tx module <NUM>, whereby the Tx module <NUM> includes a transmitter coil <NUM> and an amplifier <NUM>. In some embodiments, the transmitter coil <NUM> is a tri-axis coil generally configured to generate the EM field <NUM> at a strength, referred to herein as the transmit power, wherein the transmit power is based on electrical power provided by the amplifier <NUM> to the transmitter coil <NUM>. The amplifier <NUM> is a programmable amplifier generally configured to generate the electrical power at a magnitude based on received control signaling. Thus, the transmit power for the EM field <NUM> is a programmable value that is controlled at the base device <NUM>. The Tx module further includes a digital-to-analog DAC converter <NUM> that converts values provided by a CPU <NUM> of the base device <NUM> to supply the input to the amplifier <NUM>. In other embodiments, the amplifier <NUM> is a non-programmable amplifier, and magnitudes of the EM field can be changed simply by reducing the amplitude of the source signal, which could be done digitally or via the DAC. The CPU <NUM> can perform other operations on behalf of the base device <NUM> and the tracking system <NUM>, including generation of poses for the stylus <NUM> as described further herein. In other embodiments one or more of these operations can be performed by the CPU <NUM> or by a third CPU.

To support reading of EM data, the stylus <NUM> includes an Rx module <NUM> having a receiver coil <NUM> and an analog-to-digital converter (ADC) <NUM>. In some embodiments, the receiver coil <NUM> is a tri-axis coil configured to generate an analog electrical signal having a magnitude and/or phase indicative of a detected EM field <NUM>. In some embodiments, such as described further below with respect to <FIG>, the receiver coil <NUM> may have fewer than three coils. The ADC <NUM> is generally configured to receive the generated analog signal and convert the analog signal to a digital value indicative of the analog signal, and therefore indicative of the detected EM field <NUM>. Thus, in the example of <FIG>, the digital values generated by the ADC <NUM> are EM data that can be used for pose identification as described further below.

To support generation of poses, the stylus <NUM> includes a secondary sensor <NUM> configured to generate non-EM pose information for the stylus <NUM>. In some embodiments the secondary sensor is an inertial measurement unit (IMU). In other embodiments, the secondary sensor is part of a secondary tracking system (not illustrated at <FIG>) that generates secondary pose information based on a secondary tracking signal, such as a light-based signal (e.g., a laser, LED light, and the like), a sonic signal (e.g., an ultrasonic signal), and a radio frequency (RF) signal.

To support pose identification (that is, identification of poses of either the base device <NUM> or the mobile device <NUM>) the stylus <NUM> and the base device <NUM> include processors <NUM> and <NUM> respectively. The processors <NUM> and <NUM> are general-purpose or application-specific processors generally configured to execute instructions (e.g., computer programs) in order to carry out specified tasks. In some embodiments, at least one of the processors <NUM> and <NUM> executes instructions to identify a pose associated to the base device <NUM> or the mobile device <NUM> based on a weighted combination of the EM data provided by the ADC <NUM> and the secondary pose data generated by the secondary sensor <NUM>. In other embodiments, a different processor on an external system (such as a PC computer, for example) is used to compute pose. In one example, in some embodiments, the processor <NUM> identifies the pose based on the following formula: <MAT> where Wem is the weight applied to the pose computed from the EM signal, poseem is the pose computed from the EM data, Wsecondary is the weight applied to the pose computed from the secondary tracking signal <NUM>, and posesecondary is the pose computed from the secondary tracking signal.

In addition, in the example of <FIG> the processors <NUM> and <NUM> are connected via a communication link <NUM> to support communication of EM data, secondary pose data, identified poses, or other information. For example, in some embodiments the processor <NUM> identifies poses based on the EM data and secondary pose data and communicates the identified poses to the processor <NUM>. In other embodiments, the processor <NUM> communicates the EM data and the secondary pose data to the processor <NUM>, which identifies poses based on the weighted combination. The communication link <NUM> can be a wired communication link, a wireless communication link (e.g. Bluetooth), and the like, or a combination thereof. In other embodiments, the EM data and the secondary pose data can be sent to a third processor (not pictured) where the pose is computed from the weighted combination of the EM data and the secondary pose data.

In some embodiments, one or more of the processors <NUM> and <NUM> (or a third processor not shown at <FIG>) execute additional sets of instructions to use the poses determined based on the EM data and the secondary pose data. For example, in some embodiments the processor <NUM> can execute sets of instructions to display a virtual reality environment to a user and employ the identified poses to determine a location of the stylus <NUM> in the virtual reality environment, thereby allowing the user to interact with the virtual reality environment using the stylus <NUM>.

In some embodiments, the stylus <NUM> is used for capturing handwriting or a hand drawing. The user can write or draw (e.g. with an ink or pencil tip) on paper and the writing or drawing is captured by a computing device based on poses of the stylus <NUM>, allowing the writing or drawing to be displayed at the computing device. In other embodiments, the stylus <NUM> is used as a stylus on a computer screen in typical stylus use cases. In these embodiments, the stylus could be used to annotate, write, or as regular mouse input. In still other embodiments, the stylus <NUM> is used as a 3D mouse to manipulate 3D virtual objects. For example, the stylus <NUM> can be used instead of a "space mouse" to manipulate rotation or displacement of 3D CAD drawings.

As noted above, in some embodiments the Tx and Rx modules of the tracking system <NUM> are in reverse positions relative to <FIG>. An example is illustrated at <FIG>, which illustrates a tracking system <NUM> in accordance with some embodiments. The tracking system <NUM> includes similar components to those described above with respect to the tracking system <NUM>. However, for the tracking system <NUM> the TX module <NUM> is located at the stylus <NUM> while the Rx module <NUM> is located at the base device <NUM>. The components perform similarly to the corresponding components of <FIG> to support generation of poses for the stylus <NUM>.

It will be appreciated that the stylus <NUM> can be any kind of writing device. <FIG> illustrates an example embodiment of the stylus <NUM> that can be employed to write on paper or other media while concurrently generating poses for the stylus <NUM>. In the depicted example, the stylus <NUM> includes both the EM coil <NUM> to support generation of EM poses, and also includes writing material <NUM>. The writing material <NUM> can be any kind of writing material that is dispensed from a tip <NUM> of the stylus <NUM> onto a writing surface (e.g. paper or other medium) as a user moves the stylus <NUM>. Thus, in some embodiments the writing material <NUM> is ink contained in a reservoir, wherein both the reservoir and the EM coil are placed within a housing that forms the outside of the stylus <NUM>. In other embodiments, the stylus <NUM> is a pencil and the writing material <NUM> is graphite that is contained within a housing of the stylus <NUM> and is dispensed at the tip <NUM> of the stylus <NUM>. In other embodiments, the stylus <NUM> includes a non-writing nib configured to be placed on a computer or tablet screen or other surface.

In some cases, metal or other material situated in close proximity to the Rx module <NUM> or to the Tx module <NUM> can cause distortions in the EM field <NUM>, and these distortions can result in inaccuracies in the EM pose data. For example, if the stylus <NUM> is being used to write on a table containing metal, such that the tip <NUM> is close to the metal, the resulting distortions in the EM field <NUM> can cause errors in the EM pose data, resulting in pose tracking errors for the stylus <NUM>, and in turn resulting in a poor user experience. However, the distortions in the EM field can be mitigated by placing the EM coil <NUM> at a distance from the surface that is causing the distortion. For example, in some embodiments, including the example embodiment of <FIG>, the EM coil <NUM> is placed relatively close to an end <NUM> of the stylus <NUM>, the end <NUM> opposite the tip <NUM>. The EM coil <NUM> is therefore relatively far away from the distorting surface when the tip <NUM> is in contact with that surface, thereby reducing distortions in the EM field <NUM> and in turn improving the accuracy of the EM pose data.

In some cases, it may be desirable to reduce the cost or size of the circuitry in the stylus <NUM>. An example is illustrated at <FIG> in accordance with some embodiments. In the depicted embodiment, the stylus <NUM> and base device <NUM> are configured similarly to the embodiment of <FIG>. However, the stylus <NUM> employs an EM coil <NUM> that is wound around an axis <NUM> along the length of the stylus <NUM>, which allows the EM coil <NUM> to be placed within the housing of the stylus <NUM> with relatively small impact on the form factor of the stylus <NUM>. In some embodiments, the EM coil <NUM> is the only coil employed by the Rx module. In other embodiments, the Rx module employs EM coils in addition to the EM coil <NUM>. In addition, by winding the coil <NUM> along the length of the stylus <NUM>, the coil <NUM> can be implemented with a relatively high number of turns. As is understood in the art, the strength of the EM field <NUM> is proportional to the number of turns in the coil, such that a long coil wound around the stylus <NUM> allows for higher signal-to-noise ratio in the EM data while having a relatively small impact on the size of the stylus <NUM>.

In some embodiments, the tracking system can reduce the overall cost of the system by employing existing EM components of a computer, smartphone or other device. An example is illustrated at <FIG>. In the depicted example, the stylus <NUM> is configured similarly to the embodiment of <FIG>, including the EM Tx module <NUM> and the second sensor <NUM>. However, in the illustrated example, the base device is a smartphone <NUM> including a CPU <NUM> and a magnetometer <NUM>. In some embodiments, the magnetometer <NUM> is a <NUM>-axis magnetometer embedded in the smartphone <NUM> and used for different smartphone operations, such as providing compass-like capabilities. The CPU <NUM> is configured to control the magnetometer <NUM> to operate in similar fashion to the EM coil <NUM> of <FIG>, and in particular to collect EM pose data from readings of the EM field <NUM>. That is, by using the magnetometer <NUM> as the base coil for reading EM signals transmitted by the stylus <NUM>, the CPU <NUM> can act as the "base" and collect EM pose data from the magnetometer <NUM>, which can then be converted to EM poses by the CPU <NUM>.

In some embodiments, the magnetometer <NUM> has a relatively low sample rate (e.g. in the <NUM>-<NUM> range) which does not readily allow for frequency multiplexing of the EM fields, and accordingly the stylus <NUM> can employ a pulsed EM configuration which is better suited for these magnetometers. For smartphones with magnetometers or other magnetic sensor with higher sampling rates, EM fields can be multiplexed over different frequencies for simultaneous transmission.

It will be appreciated that the stylus <NUM>, instead of or in addition to supporting the deposition of writing material on a surface, can employ different configurations of the stylus tip. The different tips can support improved pose detection, additional stylus functionality, or a combination thereof. Examples of different stylus tip configurations are illustrated at <FIG> in accordance with some embodiments. <FIG> illustrates a stylus <NUM> including a CPU <NUM> and an EM module <NUM>. The EM module <NUM> can be a Rx EM module similar to the Rx EM module <NUM> of <FIG>, or a Tx EM module similar to the Tx EM module <NUM> of <FIG>. Accordingly, the stylus <NUM> can be employed instead of the stylus <NUM> in any of the embodiments described with respect to <FIG> above.

The stylus <NUM> employs a force sensitive tip configuration, composed of a tip <NUM>, a spring <NUM>, and a force sensor <NUM>, which can be a mechanical force sensor or an electrical force sensor. In some embodiments, the spring <NUM> is omitted. When the tip <NUM> contacts a surface (e.g. when a user places the tip <NUM> on a screen or other writing surface), the force of the surface against the tip translates down the tip applying a force that is sensed by the force sensor <NUM>, which indicates the sensed force to the CPU <NUM>. In some embodiments, the force sensor <NUM> indicates the sensed force to the CPU <NUM> via a binary signal to indicate whether the force at the tip <NUM> has crossed a specified or programmable threshold. In other embodiments, the force sensor <NUM> indicates the level of the sensed force to the CPU <NUM> via an electrical (digital or analog) signal, thereby indicating variable levels of pressure at the tip <NUM>. Based on the information provided by the force sensor <NUM>, the CPU <NUM> can take one or more specified actions, such as executing an application, changing an application parameter (such as the thickness, shade, color, or other aspect of a displayed or stored line), and the like.

In some embodiments, such as the example of <FIG>, the stylus can employ a tip that is used to assist in pose identification for the stylus. <FIG> illustrates a stylus <NUM> including a CPU <NUM> and an EM module <NUM>. As with <FIG>, the EM module <NUM> can be a Rx EM module similar to the Rx EM module <NUM> of <FIG>, or a Tx EM module similar to the Tx EM module <NUM> of <FIG>. Further, the stylus <NUM> can be employed instead of the stylus <NUM> in any of the embodiments described with respect to <FIG> above. The stylus <NUM> has a tip configuration that acts as a <NUM>-D sensor when dragged against an arbitrary flat surface, simulating a pen. The configuration includes a ball tip <NUM> and rollers and rotation sensor pairs <NUM> and <NUM>. As the user moves the stylus over the surface, the ball tip <NUM> rotates. The roller and rotation sensor pairs <NUM> and <NUM> sense the rotation of the ball tip and based on the rotation provide <NUM>-D pose information to the CPU <NUM>. For example, in some embodiments the roller and rotation sensor pair <NUM> indicates the position of the tip <NUM> along a "y" axis and the roller and rotation sensor pair <NUM> indicate the position of the tip <NUM> along a corresponding "x" axis. The <NUM>-D pose information can be used to check or augment the pose identified based on the EM pose data as described above.

In some embodiments, such as the example of <FIG>, instead of tracking the <NUM>-D position of the tip via a mechanical configuration, the stylus can employ optical sensing. <FIG> illustrates a stylus <NUM> including a CPU <NUM> and an EM module <NUM>. As with <FIG>, the EM module <NUM> can be an Rx EM module similar to the Rx EM module <NUM> of <FIG>, or a Tx EM module similar to the Tx EM module <NUM> of <FIG>. Further, the stylus <NUM> can be employed instead of the stylus <NUM> in any of the embodiments described with respect to <FIG> above. The stylus <NUM> includes a light source <NUM> and a photo sensor <NUM>. The light source <NUM> is configured to emit a light beam <NUM> via a tip of the stylus <NUM>. The light beam <NUM> is reflected off a surface <NUM> external to the stylus <NUM>, such as writing surface, computer screen, and the like. The photosensor <NUM> uses the reflected light beam to capture images of the surface <NUM> and provides the images to the CPU <NUM>. The CPU <NUM> compares each captured image to previous captured images to determine a direction of movement of the stylus across the surface <NUM>. The CPU <NUM> can employ the identified direction of movement to augment or check the pose identified based on the EM pose data.

It will be appreciated that any of the stylus embodiments described above, including the styli <NUM>, <NUM>, <NUM>, and <NUM>, can include additional components to provide additional inputs or other information to a computer device. For example, the stylus can include one or more buttons to indicate additional input data to the computer device. The one or more buttons can be placed in convenient location for the user such as, if the stylus is held like a pen or pencil, one of the buttons is placed at or near the user's index finger while another button is placed at or near the user's thumb. In some embodiments, a button can be placed at an end of the stylus opposite the tip, in a similar location as the button for a retractable ball point pen. Additionally, the stylus can incorporate, in some embodiments, a slider, <NUM>-D trackpad, roller, or similar component (or a combination thereof) that allows for <NUM>-dimensional linear input. In some embodiments, the <NUM>-D linear input component is placed near the front of the stylus in the location where, when held as a pen, the index finger would normally be placed. In other embodiments, the component is placed closer to the middle for easy access when the stylus is held as a wand. In other embodiments, the stylus can include a <NUM>-D trackpad, joystick, or similar component that allows for <NUM>-D analog input into the stylus. In some embodiments, the <NUM>-D analog input component is placed closer to the middle for easy access with the thumb when the stylus is held as a wand.

In some embodiments, the flexibility and portability of the stylus can be enhanced by allowing the base device <NUM> to be removably integrated with the stylus. An example is illustrated at <FIG> in accordance with some embodiments. <FIG> illustrates a stylus <NUM>, which in different embodiments corresponds to any of the styli <NUM>, <NUM>, <NUM>, and <NUM> described above, or a combination thereof. <FIG> also illustrates a dongle <NUM> including an EM module <NUM>. The EM module <NUM> can be an Rx EM module similar to the Tx EM module <NUM> of <FIG>, or an Rx EM module similar to the Tx EM module <NUM> of <FIG>. The stylus <NUM> includes an EM module <NUM> that is complementary to the EM module <NUM>. Thus, if the EM module <NUM> is a Tx module then the EM module <NUM> is an Rx module and vice versa. The dongle <NUM> can therefore be mated with the connector <NUM> so that it is easy and convenient for the user to carry the dongle <NUM> with the stylus <NUM>. In some embodiments, the stylus <NUM> provides power from a battery (not shown) or other power source to the dongle <NUM> via the connector <NUM> to charge a battery (not shown) that provides power to the EM module <NUM>.

The dongle <NUM> can be plugged into a USB port of a computer device, such as smartphone <NUM> at USB port <NUM>. While plugged into the USB port <NUM> the EM module <NUM> is active, either transmitting or receiving the EM field <NUM>, allowing the smartphone <NUM> to identify poses based on EM pose data as described above with respect to <FIG>. In some embodiments the dongle <NUM> itself includes a processor to identify pose data, reducing processing load at the smartphone <NUM>. It will be appreciated that other configurations and form factors for the dongle <NUM> are possible. For example, in some embodiments the dongle <NUM> has a "cap" form factor that can fit over the tip of the stylus <NUM>.

In some embodiments, the EM pose data can be used to assist in handwriting or other text or drawing identification (referred to collectively herein as "handwriting capture"). For example, in one embodiment, handwriting that is captured via the EM pose data is converted (e.g. by one or both of the processors <NUM> and <NUM>, or by another processor) from handwriting to typed text through use of handwriting recognition software or optical character recognition (OCR) software. In another embodiment, the handwritten capture is converted to geometric shapes through the use of shape recognition software. Such processing can "clean up" hand-drawn diagrams or generate readable variants of charts and graphs. In another embodiment, the handwritten capture is converted to mathematical equations. Such processing can "clean up" math equations into presentable digital representations.

Using EM pose data allows for handwriting capture on a variety of surfaces, in contrast to conventional handwriting capture that requires use of a specialized surface. In particular, using EM pose data supports handwriting capture when the relative angle or orientation of the surface is unknown. For example, the user could write on a wall (where the surface is vertical) and also write on a desk (where the surface is horizontal) and the angle of the writing plane is not known a-priori.

To support handwriting capture on an unknown surface, in some embodiments the handwriting capture from stylus <NUM> is "fit" onto a plane to produce an accurate representation of the handwriting. For example, if the writing surface were angled at <NUM> degrees but the trajectories are projected onto a plane that is flat (angled at zero degrees), the resulting handwriting would look distorted. In order to preserve an accurate representation of what was handwritten, the plane of the writing surface is determined by a processor, such as the processor <NUM> or the processor <NUM>. In one embodiment, the processor employs a plane-fitting algorithm to "fit" the writing trajectories to determine the plane angle. In other embodiments, the writing plane is determined "on the fly" based on what the user is writing. In other embodiments, the plane fitting is done with post-processing at the processor.

In some embodiments, the processor determines the angle of writing is based on the direction of the writing. An example is illustrated at <FIG>, which illustrates handwriting <NUM> that has been written on a surface with the stylus <NUM>. The processor <NUM> (or other processor) employs a heuristic to determine an angle <NUM> relative to a vector <NUM> that represents "horizontal" writing (e.g., left to right or right to left). The processor <NUM> uses the angle <NUM> to correct the orientation of the handwriting prior to handwriting recognition. For example, in some embodiments the processor <NUM> fits a line <NUM> through the handwriting and measures the angle <NUM> between the fitted line and the desired angle. In other embodiments the orientation of the handwriting can be determined by a machine learning model, such as a neural network.

In some embodiments, the processor performs orientation fitting. For example, handwriting recognition software that can recognize handwriting can incur errors if the handwriting were rotated <NUM> degrees so that the handwriting is upside-down. Thus, before attempting to recognize the handwriting, the processor <NUM> determines the "correct side up" orientation of the writing. In some embodiments, the processor attempts to rotate the handwriting by some amount, and then uses a heuristic to score the desirability of the resulting rotation. The processor repeats the process until the score crosses a threshold. An example is illustrated at <FIG> in accordance with some embodiments. At block <NUM>, the processor <NUM> gives the handwriting an initial orientation and an initial score based on the initial orientation. For example, in some embodiments the score can vary in a range of <NUM> to <NUM> and represents a confidence level that the handwriting is in a horizontal orientation. At block <NUM>, the processor <NUM> determines if the score exceeds a threshold. In the above example, the threshold can be a value of <NUM>, indicating a threshold confidence level that the orientation of the handwriting is such that the handwriting can be interpreted by a specified handwriting recognition algorithm. If the score does not exceed the threshold, the method moves to block <NUM> and the processor <NUM> rotates the orientation of the handwriting by a specified amount. The method returns to block <NUM> and the processor <NUM> generates a new score based on the adjusted orientation. The processor <NUM> continues to incrementally adjust the orientation until, at block <NUM>, the score exceeds the threshold. In response to the score exceeding the threshold, the method flow moves from block <NUM> to block <NUM>, and the processor <NUM> executes a handwriting recognition algorithm based on the current orientation of the handwriting (the orientation that resulted in the score exceeding the threshold.

In other embodiments, the processor <NUM> rotates the captured handwriting by multiple angles, and each rotation is processed in parallel to speed up the processing. In some embodiments, the scoring is performed by attempting handwriting recognition of some of the handwriting and using the level of confidence from the handwriting recognition algorithm as an input into the score. In yet other embodiments, a machine learning model is employed to estimate the angle of rotation necessary for handwriting recognition software to properly process the handwriting strokes.

Embodiments of the present disclosure include fusing of multiple data sources to improve the tracking quality of the pen device. In one embodiment, the poses from the EM position tracking subsystem are combined with poses from an IMU to reduce noise and improve angular accuracy. In some embodiments a Kalman filter is used to combine data derived from an IMU with data derived from the EM position tracking system to produce a "fused" pose. Fusion can also be used to correct for temporary distortions caused by nearby metal, for example by "dead reckoning" for short periods of time with the IMU when metallic distortion is detected.

Claim 1:
A writing device comprising:
an electromagnetic, EM, tracking module (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) to measure a magnitude and/or phase of a generated EM field (<NUM>) and to generate, using measured magnitude and/or phase field values, EM pose data indicative of a relative pose between an EM transmitter (<NUM>) and an EM receiver (<NUM>, <NUM>);
a processor (<NUM>) to process the EM pose data for identification of a pose of the writing device (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) based on the relative pose;
a sensor (<NUM>) to generate a first sensor pose of the writing device based on sensor data;
characterized in that
the processor is configured to identify the pose of the writing device based on a weighted combination of the relative pose and the first sensor pose.