Patent Description:
Currently, several systems or methods exist for recognizing the handwriting of a user. One type of handwriting recognition, called online handwriting recognition, consists of performing the recognition of a character, or a sequence of characters, while the user is writing it. Another type of handwriting recognition, called offline handwriting recognition, is based on the analysis of a picture showing text.

For example, it is know from patent document <CIT> to perform online handwriting recognition by using a device such as a mobile terminal and three sensors being an accelerometer, a gyroscope and a magnetometer. The method described in this document uses a neural network of BLSTM type (Bidirectional Long-short term memory). The data acquired by the three sensors are submitted to several pre-processes in order to be analyzed by the neural network. Moreover, the neural network is trained with a dictionary containing predetermined words. The neural network is trained such that it is able to recognize the beginning and the end of a word being written and, on this basis, the neural network is able to determine which word of the dictionary has been written. One drawback of this method is then the use of three sensors which leads to a device expensive to build. Another drawback of this document is that the neural network is trained to recognize predetermined words. The recognition is not made continuously.

It is also known from the scientific publication <NPL>, a method for detecting online handwriting. The method uses four neural networks: MC-FCRN (multi-spatial-context fully convolutional recurrent network), FCN (fully convolutional network), LSTM (long-short term memory) and CTC (connectionist temporal classification). The neural networks are jointly trained. The method uses the path signature method, applied on reduced windows of data. This publication does not describe how the data are effectively acquired, however using path signature method implies to use positional sensors like a camera or a touchscreen. Another drawback is the need of using four neural networks, each being trained in chain with the entry of the preceding neural network.

It is also known from the publication <NPL>, a method for offline handwriting recognition which cut line-of-text images into small time-steps before feeding them in a feature extractor neural network. However, this publication performs offline handwriting recognition, which is not adapted for the online handwriting recognition. Another example of relevant prior art is <NPL>.

One purpose of this disclosure is to improve the situation.

It is now referred to <FIG> illustrating embodiments of a system <NUM> for recognizing online handwriting. The same reference numbers are used to describe identical elements of the system <NUM>.

<FIG> generally illustrates a system <NUM> according to a first embodiment. The system <NUM> comprises a handwriting instrument <NUM>. Typically, the handwriting instrument <NUM> can be a pen, a pencil, a brush or any element allowing a user to write with it on a support. Typically, the support can be paper, canvas, or any surface on which a user can write or draw.

The handwriting instrument <NUM> comprises a body <NUM> extending longitudinally between a first end <NUM> and a second end <NUM>. The first end <NUM> comprises a writing tip <NUM> which is able to write on a support. Typically, the tip <NUM> can deliver ink or color.

The system <NUM> may comprise a module <NUM>. The module <NUM> comprises at least one motion sensor <NUM>. In one embodiment, the motion sensor <NUM> can be a three-axis accelerometer.

In another embodiment, the module <NUM> may further comprise a second motion sensor <NUM>. For example, the second motion sensor <NUM> is a three-axis gyroscope.

In an embodiment, the module <NUM> is embedded in the handwriting instrument <NUM>. For example, the module <NUM> is embedded near or at the second end <NUM> of the handwriting instrument.

In another embodiment, the module <NUM> is distinct from the handwriting instrument <NUM>. In this embodiment, shown <FIG>, the module <NUM> is able to be placed on the second end <NUM> of the handwriting instrument.

Hence, the module <NUM> may comprise a hole of a diameter D allowing the insertion of the second end <NUM> of the handwriting instrument in it.

In these configurations, at least one motion sensor <NUM> is then placed or at the second end <NUM> of the handwriting instrument <NUM>. In this way, the user of the handwriting instrument is not bothered when he uses the handwriting instrument <NUM>, since module <NUM> does not block his sight.

For all the embodiments described hereinafter, the module <NUM> can be embedded in the handwriting instrument <NUM>, or distinct from the handwriting instrument <NUM>.

In all the embodiments, at least one motion sensor <NUM> is able to acquire data on the handwriting of the user when the user is using the handwriting instrument <NUM>. These data are communicated to a calculating unit <NUM> which is configured to analyze the data and perform the recognition of handwriting.

The calculating unit <NUM> can comprise a volatile memory to store the data acquired by the motion sensor <NUM> and a non-volatile memory to store a machine learning model enabling the handwriting recognition.

In the embodiment shown <FIG>, the module <NUM> further comprises the calculating unit <NUM>.

In another embodiment shown <FIG>, the module <NUM> does not comprise the calculating unit <NUM>.

In this embodiment, the calculating unit <NUM> may be comprised in a mobile device. The mobile device <NUM> can typically be an electronic tablet, a mobile phone or a computer.

The module <NUM> can also comprise a short-range radio communication interface <NUM> allowing the communication of data between the at least one motion sensor <NUM> and the calculating unit <NUM>. In one embodiment, the short-range radio communication interface is using a Wi-Fi, Bluetooth®, LORA®, SigFox® or NBIoT network. In another embodiment, it can also communicate using a <NUM>, <NUM>, <NUM> or <NUM> network.

The mobile device <NUM> further comprises a short-range radio communication interface <NUM> enabling communication between the calculating unit <NUM> and the module <NUM> via the short-range radio communication interface <NUM>.

In another embodiment, the calculating unit <NUM> may be comprised in the cloud. The calculating unit <NUM> may be accessible via the mobile device <NUM>.

When the calculating unit <NUM> is comprised in the module <NUM>, as shown on <FIG>, the module <NUM> may still comprise the short-range radio communication interface <NUM>, in order to communicate results to a deported server (not shown), as a non-limitative example.

The module <NUM> may further include a battery <NUM> providing power to at least the motion sensor <NUM> when the user is using the handwriting instrument. The battery <NUM> can also provide power to all the other components that may be comprised in said module <NUM>, such as the calculating unit <NUM>, the short-range radio communication interface <NUM>.

In all the embodiments described above, the calculating unit <NUM> receives motion data acquired from at least on motion sensor <NUM> to analyze them and perform handwriting recognition.

In a preferred embodiment, the motion data received by the calculating unit <NUM> are raw motion data.

More specifically, the calculating unit <NUM> can store a machine learning model able to analyze the motion data acquired by the motion sensor <NUM>. The machine learning model can comprise at least one neural network. The machine learning model is trained to perform the recognition of a sequence of characters which has been written by the user while using the handwriting instrument <NUM>.

The machine learning model is trained in a multi-task way.

More specifically, the machine learning model is trained to perform two tasks being the recognition of a character which has been written by the user while using the handwriting instrument <NUM>, which is designated as the first task, and the segmentation of strokes, which is designated as the second task.

The first and second tasks share the same backbone being a neural network, which extracts the hidden features of the motion data. This neural network is, in a non-limitative embodiment, a convolutional neural network such as an encoder. This encoding neural network is designated hereinafter as the shared backbone.

The first task may be performed, in a non-limitative embodiment, by the neural network being the shared backbone, together with a recursive neural network, such as a BLSTM or a Transformer, trained with the CTC loss function.

The second task may be performed, in a non-limitative embodiment, by an auto-encoder neural network, whose encoder is the shared backbone and decoder is an additional convolutional neural network with upsampling.

The machine learning model is trained in a multi-task way such that it is capable of performing at least these two tasks at the same time.

Reference is made to <FIG>, illustrating steps of the training phases of the multi-task machine learning model.

In step S1, the calculating unit <NUM> receives the motion data acquired by at least one motion sensor <NUM> while the user is writing a character, or a sequence of characters with the handwriting instrument <NUM>.

A step S2 of pre-processing is performed. The pre-processing step comprises the time windowing of the motion data such that the motion data is cut into time-steps.

For example, the motion data is windowed into time-steps of a size comprised between <NUM> and <NUM> seconds.

The windowed motion data, i.e. the time-steps, are then fed in the shared backbone. The shared backbone further receives labels at step S3.

In an embodiment, the labels correspond to on-paper and in-air time stamped labels. In other words, the labels correspond to whether the writing tip <NUM> of the handwriting instrument <NUM> touched the paper, which corresponds to the on-paper label, or not, which corresponds to the in-air label. The labels are time stamped and relative to each data sample.

Typically, when using a motion sensor <NUM> such as a three-axis accelerometer (or a three-axis accelerometer and a three-axis gyroscope), the motion data acquired by the motion sensor <NUM> is continuous and lacks positional information. More particularly, the motion data comprise both motion data corresponding to when the user is writing with it on a support and motion data corresponding to when the user is just moving the handwriting instrument <NUM> in the air, without writing.

At step S5, the shared backbone performs hidden features extraction to obtain intermediate features at step S6.

From these intermediate features, the segmentation task, or second task, is performed at step S7. This step allows finding the starting and ending time of each stroke. During this step, the segmentation task comprises the segmentation of the on-paper strokes and in-air movements.

At step S8, the on-paper strokes and in-air movements classes are obtained for each time sample in the signals.

The intermediate features of step S6 are also used in step S9. In step S9, the machine learning model performs the concatenation in time sequence of the on-paper strokes and in-air movements.

At step S10, other labels are sent to the machine learning model for performing the first task of sequence of characters recognition.

More specifically, these labels may be fed into the recursive neural network participating in the first task.

These labels may correspond to character labels or character sequence labels that are used in the character sequence classification step S11.

The labels may be individual characters, words, or even sentences.

The classification step S11 may be performed at least in part by the recursive neural network, such as a BLSTM or a Transformer, trained with a loss function such as the CTC loss function, as described above.

Finally, a characters sequence is obtained at step S12, which corresponds to the characters sequence written by the user while using the handwriting instrument <NUM>, and to which the motion data was associated at step S1.

During the training of the machine learning model, the machine learning model learns to recognize both on-paper strokes and in-air movements from the motion data in addition to continuously recognize characters sequence.

The multi-task training of the machine learning model renders the system more efficient. Indeed, the learning of the second task and the first task, at the same time, makes the learning of the first task significantly more efficient.

Moreover, since the machine learning model acquires the knowledge to identify individual on-paper strokes and in-air movements from the motion data, it is able to learn any character of any language.

Steps S1 to S6 and S9 to S12 may be assimilated to the first task of characters sequence recognition.

Steps S1 to S8 may be assimilated to the second task of stroke segmentation.

Once the multi-task training is performed, the machine learning model can be stored in the calculating unit <NUM>.

In an embodiment, only the weights of the first task are used during the inference phase, such that the machine learning model stored in the calculating unit only comprises the weights of the first task.

In another preferred embodiment, the machine learning model is further trained to differentiate forward in-air movement and backward in-air movement. Indeed, paper strokes are not always ordered such that it cannot be assumed that the most recent stroke belongs to the most recent character, as people can go back to finish a previous character.

This is for example illustrated on <FIG> and <FIG> showing the characters sequence "written down" being written on paper and the associated motion data.

In this example the motion data corresponds to the acceleration signal acquired by a three-axis accelerometer. The graphic of <FIG> shows the amplitude of the acceleration signal according to the tree-axis x, y, z as a function of time.

The characters sequence of <FIG> comprises ten paper stroke, six forward jumps between letters, one forward jump between words and two backward jumps (before the "i" point and before the "t" bar).

The circled numbers are associated with each of the paper stroke and show the order in which they have been written.

It then appears that it may be important to consider also the direction of the in-air movements.

In particular, the detection of forward in-air movement may help the machine learning model to recognize the space between characters or words. The detection of backward in-air movement may help the machine learning model to recognize that a character is being finished (for example with the points on the i).

Then, in step S3 of <FIG>, the labels can include on-paper time label, forward in-air movement and backward in-air movement, such that in step S7 the segmentation task is made to obtain on-paper stroke, forward in-air movement and backward in-air movement.

In another embodiment, the machine learning model is further trained with another label. This label may be other in-air movement.

This label can help the machine learning model to determine that a user is not writing anymore. For example, it detects that the user is thinking or simply wagging the handwriting instrument <NUM> in the air.

The trained machine learning model can then be stored in the calculating unit <NUM>.

<FIG> illustrates the inference phase of the machine learning model.

At step S20, the motion data is obtained from at least one motion sensor <NUM>.

Preferably, the motion data corresponds to the raw signals acquired by at least one motion sensor <NUM> while the user is writing a sequence of characters with the handwriting instrument <NUM>.

The motion data are then pre-processed by windowing these motion data into time steps at step S21. This step corresponds to step S2 described in reference with <FIG>.

The windowed motion data are then fed, at step S22, in the shared backbone which performs the features extraction such as described with reference to step S5 of <FIG>.

From these features extraction, the machine learning model performs the concatenation in time sequence, such that the character sequence classification performed at step S24 is made on the on-paper strokes and in-air movement in the correct chronological order.

As already described, the in-air movement may comprise forward in-air movement and backward in-air movement.

At step S25, a character sequence is obtained as an output of the machine learning model, which corresponds to the character sequence written by the user with the handwriting instrument.

During the inference phase, the machine learning model does preferably not perform the segmentation task anymore since the shared backbone has already been trained with the weights corresponding to both the segmentation and the character classification task, due to the multi-task training.

Claim 1:
Method for recognizing online handwriting comprising
- acquiring, by means of a handwriting instrument (<NUM>) comprising a module (<NUM>) comprising at least one motion sensor (<NUM>), motion data on the handwriting of the user when said user is writing a sequence of characters with said handwriting instrument (<NUM>), said handwriting instrument (<NUM>) further including a body (<NUM>) extending longitudinally between a first end (<NUM>) and a second end (<NUM>), said first end (<NUM>) having a writing tip (<NUM>) which is able to write on a support,
- analyzing said motion data with a machine learning model being configured to deliver as an output the sequence of characters which was written by the user with said handwriting instrument (<NUM>),
characterized in that the machine learning model is trained in a multi-task way,
the method further comprising a prior step of multi-task training of the machine learning model, wherein the machine learning model is trained to perform:
- a stroke segmentation task,
- a character classification task.