System, method, and apparatus for triggering recognition of a handwritten shape

A technique that uses repetitive and reliably recognizable parts of handwriting, during digital handwriting data entry, to trigger recognition of digital ink and to repurpose handwriting task area properties. In one example embodiment, this is achieved by drawing one or more delayed strokes of a desired sub-word unit using a stylus on a touch screen. An associated data of the drawn one or more strokes is inputted via the touch screen into a handwriting recognition engine. A first trigger stroke in the drawn one or more strokes that can be used to trigger the sub-word unit recognition by the handwriting recognition engine is then determined. The sub-word unit recognition is then triggered for the drawn one or more strokes based on the determined first trigger stroke by the handwriting recognition engine.

TECHNICAL FIELD OF THE INVENTION

The present invention relates generally to a user interface in a computer system, and more particularly relates to triggering handwriting recognition during data entry into a computer.

BACKGROUND OF THE INVENTION

The character recognition approach requires writing characters using a pen to naturally write an entire character on a graphics tablet. In this approach, the character recognition technique attempts to find the character that most closely matches the strokes entered on the tablet and returns the results. The faster the results are returned, the better a user's experience will be. Conventional character recognition techniques typically have a fundamental bottleneck in detecting when the user finishes writing. This problem is even more severe when writing characters of scripts, such as Indic, Arabic, South-east Asian, and the like, which have complex shapes and can require writing these shapes using multiple strokes.

Conventional ways for detecting that a user has finished writing and that character recognition should be triggered typically include one or more of the following techniques:

First, a timer is set upon detecting that the user has lifted the pen off the character recognition surface, such as the graphics tablet, which be referred to as a pen-up event. If a pen-down event, which occurs when a user places the pen on the character recognition surface, is not detected before the timer expires, it is inferred that the user has finished writing. The problem with this technique is preferred timer values can vary from user to user depending on the users' writing speeds.

Second, if the user touches outside of a current character recognition task area or inside a next task area, the pen-down event is interpreted as completion of the handwriting to be recognized from the current task area. The problem with this technique is it does not take advantage of the time between when the user lifts the pen from the current task area and when the user puts the pen down in the next task area.

Third, if the user presses a soft key, labeled “end” or the like, the button-pressed event is interpreted as an indication from the user that the user is finished writing and that recognition should be triggered. This technique reduces the quality of the user's experience by requiring many button presses. The repeated hand movement associated with these button presses can get tiresome, and can be a significant overhead from the user's point of view, and can disturb the user's train of thought in composing sentences.

The above conventional techniques, for determining when a user has stopped writing in a character recognition task, can have even more shortcomings, when writing characters of scripts, such as Indic, Arabic, South-east Asian, and the like, which have complex shapes and can require multiple strokes to write a single character or a sub-word unit.

SUMMARY OF THE INVENTION

According to an aspect of the subject matter, there is provided a method for triggering handwriting recognition upon completion of a shape entry into a computer, the method including the steps of drawing one or more strokes of a desired shape using a stylus on a touch screen, inputting an associated data of the one or more strokes via the touch screen into a handwriting recognition engine, determining a trigger stroke in the drawn one or more strokes that can be used to trigger shape recognition by the handwriting recognition engine, and triggering shape recognition for the drawn shape based on the determined trigger stroke by the handwriting recognition engine.

DETAILED DESCRIPTION OF THE INVENTION

The term “sub-word unit” in Devanagari script refers to an “akshara” or “samyukthakshara”. The term “sub-word” unit refers to a member of alphabetic, legographic, and/or phonetic/syllabic character set, which includes syllables, alphabets, numerals, punctuation marks, consonants, consonant modifiers, vowels, vowel modifiers, and special characters, and/or any combination thereof. The choice of the sub-word unit can depend on the language itself. For example, languages with a very well defined and small number of syllables may benefit from syllable sized unit.

FIG. 1illustrates an example method100of triggering handwriting recognition when entering a sub-word unit into a computer. At step110, this example method100begins by drawing one or more strokes of a desired sub-word unit using a stylus on a touch screen. In some embodiments, the step110begins by drawing one or more strokes of a desired shape using a stylus on a touch screen.

At step115, an associated data of the drawn one or more strokes are inputted into a handwriting recognition engine via a touch screen. At step120, a first trigger stroke in the drawn one or more strokes that can be used to trigger the sub-word unit recognition by the hand writing recognition engine is determined. The first trigger stroke signifies the completion of the writing of the sub-word unit and that detection can be used to trigger the handwriting recognition engine. In some embodiments, the step115includes dynamically inputting an associated data of the one or more strokes of the desired shape via the touch screen into the handwriting recognition engine. Also in these embodiments, the step120includes determining the trigger stroke in the drawn one or more stokes that can be used to trigger shape recognition by the handwriting recognition engine. In some embodiments, an associated data of the multiple strokes of a desired shape is substantially simultaneously inputted into the handwriting recognition engine.

In some embodiments, the handwriting recognition engine computes stroke recognition characteristics of each of the drawn one or more strokes with reference to a reference line. The first trigger stroke is then determined based as a function of the computed stroke recognition characteristics of each of the drawn one or more strokes. The stroke recognition characteristics can include stroke characteristics, such as aspect ratio, slope, and the like. In some embodiments, the stroke recognition characteristics are computed dynamically upon completion of drawing each stroke of a character, a sub-word unit and/or a shape.

At125, sub-word unit recognition is triggered for the drawn one or more strokes based on the determined first trigger stroke by the handwriting recognition engine. At130, a first candidate sub-word unit is produced upon triggering the sub-word unit recognition by the handwriting recognition engine. In some embodiments, the one or more trigger strokes in the drawn multiple strokes that can be used to trigger shape recognition are determined by the handwriting recognition engine. The shape recognition for the drawn shape is then triggered based on the determined one or more trigger strokes by the handwriting recognition engine.

At135, the first candidate sub-word unit is outputted. The outputting of the first candidate shape can include transmitting the first candidate shape, printing the first candidate shape, displaying the first candidate shape, and so on. In some embodiments, a candidate shape is produced upon triggering the shape recognition by the handwriting recognition engine. In these embodiments, the candidate shape is outputted.

Referring now toFIG. 2, there is illustrated a screen shot200of a drawn sub-word unit220and displaying of a candidate sub-word240produced by the handwriting recognition engine. The screen shot200shown inFIG. 2shows a handwriting task area210on a display screen205of a display device. The pen used for drawing the sub-word unit220in the handwriting task area210can be a stylus and the like. The handwriting task area210can be a touch screen and the like. Also shown inFIG. 2, is a reference line215that is used to compute stroke recognition characteristics of each drawn stroke. The stroke recognition characteristics, such as slope and/or aspect ratio of each drawn stroke is computed with respect to the reference line215

It can be seen inFIG. 2that the drawn sub-word unit220, of a Devanagari akshara, requires four strokes, which includes a last stroke to draw a head-line230to complete the drawing of the sub-word unit220. Upon completion of the drawing of the four strokes including the last stroke230, i.e., the completion of the drawing of the trigger stroke, a handwriting recognition engine determines the drawn last stroke230as the trigger stroke based on the stroke recognition characteristics determined for each drawn stroke. The recognition of the drawn sub-word unit220is then triggered by the handwriting recognition engine upon determining the trigger stroke230. The candidate shape240of the Devanagari akshara is then produced by the handwriting recognition engine upon triggering the sub-word unit recognition. As shown inFIG. 2, the produced candidate shape240is then displayed on a display screen205.

At140, the method determines whether there are one or more delayed strokes in the desired sub-word unit that need to be drawn. Based on the determination at140, the method100goes to step145if there are no delayed strokes that needs to be drawn to complete the drawing of the desired sub-word unit. At145, the method100determines whether there is another sub-word unit that needs to drawn. Based on the determination at145, the method100goes to110and repeats the steps110-140if there is another sub-word unit that needs to be drawn. Based on the determination at145, the method goes to step190and stops drawing sub-word units if there are no other sub-word units that need to be drawn.

Based on the determination at140, the method100goes to step150if there are one or more delayed strokes that needs to be drawn for the desired sub-word unit. At150, one or more delayed strokes of the sub-word unit are then drawn using the stylus on the touch screen. At155, an associated data of the drawn delayed strokes of the sub-word unit is inputted into the handwriting recognition engine. At160, a second trigger stroke that can be used to trigger the sub-word unit recognition is determined using the drawn one or more delayed strokes. At165, the sub-word unit recognition is triggered for the drawn one or more delayed strokes based on the determined second trigger stroke by the handwriting recognition engine.

At170, a second candidate sub-word unit is produced upon triggering the sub-word unit recognition by the character recognition engine. At175, the second candidate sub-word unit is outputted. In some embodiments, the second candidate sub-word unit is displayed on the display screen205. At180, the displayed second candidate sub-word unit is selected by touching the displayed candidate sub-word unit on the display screen205using the stylus on the screen.

Referring now toFIG. 3, there is illustrated a screen shot300of a drawn sub-word unit310requiring a delayed stroke330and displaying of the candidate sub-word340, including the delayed stroke, produced by the handwriting recognition engine. The screen shot300shown inFIG. 3shows the handwriting task area210on the display screen205of a display device. The pen used for drawing the sub-word unit310can be a stylus and the like. The handwriting task area210can be a touch screen and the like.

It can be seen inFIG. 3that the drawn sub-word unit310of a Devanagari akshara requires 9 strokes, which includes drawing two head-line strokes, a first head line stroke320and a second head line stroke330(delayed head-line stroke), to complete the drawing of the sub-word unit310. Upon completion of the drawing of the two head-line strokes, the first head line stroke320and the delayed stroke330, i.e., the completion of the drawing of the first and second trigger strokes, respectively, a handwriting recognition engine determines the drawn the second trigger stroke330. Recognition of the drawn sub-word unit310is then triggered by the handwriting recognition engine upon determining the second trigger stroke330. The candidate shape340of the Devanagari akshara is then produced by the handwriting recognition engine upon triggering the sub-word unit recognition. As shown inFIG. 3, the produced candidate shape340is then displayed on the display screen205.

Referring now toFIG. 4, there is illustrated a screen shot400of a drawn sub-word unit410requiring drawing a head line stroke420and two delayed strokes440and445and displaying of the candidate sub-words430and450, produced by the handwriting recognition engine. The screen shot400shown inFIG. 4shows the handwriting task area210on the display screen205of a display device. The pen used for drawing the sub-word unit410can be a stylus and the like. The handwriting task area210can be a touch screen and the like.

It can be seen inFIG. 4that the drawn sub-word unit410of a Devanagari akshara requires 7 strokes, which includes drawing a head-line stroke420and two delayed strokes440and445, to complete the drawing of the sub-word unit410. Upon completion of the drawing of the head-line stroke420, a handwriting recognition engine determines the drawn head-line stroke, i.e., the first trigger stroke420. The recognition of the drawn sub-word unit410is then triggered by the handwriting recognition engine upon determining the first trigger stroke420. The first candidate shape430is then produced by the handwriting recognition engine upon triggering the sub-word unit recognition. As shown inFIG. 4, a first produced candidate sub-word unit430is then displayed on the display screen205, since the sub-word unit410requires drawing two delayed strokes440and445; the displayed first candidate sub-word unit430is incorrect. In these embodiments, the user can either draw the required delayed strokes440and445to produce a desired second candidate sub-word unit450or the handwriting recognition engine can be designed to automatically produce the desired second sub-word unit450as shown inFIG. 4.

Referring now toFIG. 5, there is illustrated a screen shot500of a drawn sub-word unit510, requiring drawing two head line strokes520and540and two delayed strokes545and547, and displaying of the candidate sub-words530and550, produced by the handwriting recognition engine. The screen shot500shown inFIG. 5shows the handwriting task area210on the display screen205of a display device. The pen used for drawing the sub-word unit510can be a stylus and the like. The handwriting task area210can be a touch screen and the like.

It can be seen inFIG. 5that the drawn sub-word unit510of a Devanagari akshara requires 10 strokes, which includes drawing the two head-line strokes520and540and the two delayed strokes545and547, to complete the drawing of the sub-word unit510. Upon completion of the drawing of the first head-line stroke520, a handwriting recognition engine determines the drawn first head-line stoke, i.e., the first trigger stroke520, based on computed stroke recognition characteristics. The recognition of the drawn sub-word unit510is then triggered by the handwriting recognition engine upon determining the first trigger stroke520. A first candidate shape530is then produced by the handwriting recognition engine upon triggering the sub-word unit recognition. As shown inFIG. 5, the produced candidate shape530is then displayed on the display screen205.

Since the sub-word unit510shown inFIG. 5requires drawing two delayed strokes545and547, the displayed first candidate sub-word unit530is incorrect. In these embodiments, the user draws the required delayed strokes545and547and then followed by drawing the second head-line stroke540. Upon completion of the drawing of the second head-line stroke540, the handwriting recognition engine determines the drawn second head-line stoke540, i.e., the second trigger stroke540, based on computing stroke recognition characteristics with reference to a reference line215. The recognition of the drawn sub-word unit510is then again triggered by the handwriting recognition engine upon determining the second trigger stroke540. A second candidate shape550is then produced by the handwriting recognition engine upon triggering the sub-word unit recognition. As shown inFIG. 5, the produced second candidate shape550is then displayed on the display screen205.

Referring now toFIG. 6, there is illustrated a screen shot600of drawing a character in a character cluster-level entry format. All of the above-described character word-level entries with reference toFIGS. 1-5are applicable to the character-level entry shown inFIG. 6. As shown inFIG. 6, the display screen605includes first, second, and third handwriting task areas610,620, and630. It can be envisioned that the handwriting task areas can continue beyond the shown first, second and third handwriting task areas610,620, and630to enter desired shapes, words and/or numbers required in a cluster-level entry, such as an application form. It can be seen inFIG. 6that upon completion of drawing the head-line strokes625and645of a desired Devanagari aksharas615and640in the first and second handwriting task areas610and620, the handwriting recognition engine sequentially determines the associated triggered strokes and produces candidate first and second Devanagari aksharas635and650, respectively.

At185, the method100determines whether there is another sub-word unit that needs to be drawn. Based on the determination at185, the method goes to step110, if there is another sub-word unit that needs to be drawn. Based on the determination at185, the method100goes to step190, if the are no other sub-word units that needs to be drawn. At190, the method100stops drawing the sub-word unit.

Although the flowchart100includes steps110-190that are arranged serially in the exemplary embodiments, other embodiments of the subject matter may execute two or more steps in parallel, using multiple processors or a single processor organized as two or more virtual machines or sub-processors. Moreover, still other embodiments may implement the steps as two or more specific interconnected hardware modules with related control and data signals communicated between and through the modules, or as portions of an application-specific integrated circuit. Thus, the exemplary process flow diagrams are applicable to software, firmware, and/or hardware implementations.

Various embodiments of the present subject matter can be implemented in software, which may be run in the environment shown inFIG. 7(to be described below) or in any other suitable computing environment. The embodiments of the present subject matter are operable in a number of general-purpose or special-purpose computing environments. Some computing environments include personal computers, general-purpose computers, server computers, hand-held devices (including, but not limited to, telephones and personal digital assistants (PDAs) of all types), laptop devices, multi-processors, microprocessors, set-top boxes, programmable consumer electronics, network computers, minicomputers, mainframe computers, distributed computing environments and the like to execute code stored on a computer-readable medium. The embodiments of the present subject matter may be implemented in part or in whole as machine-executable instructions, such as program modules that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like to perform particular tasks or to implement particular abstract data types. In a distributed computing environment, program modules may be located in local or remote storage devices.

FIG. 7shows an example of a suitable computing system environment for implementing embodiments of the present subject matter.FIG. 7and the following discussion are intended to provide a brief, general description of a suitable computing environment in which certain embodiments of the inventive concepts contained herein may be implemented.

A general computing device, in the form of a computer710, may include a processing unit702, memory704, removable storage701, and non-removable storage714. Computer710additionally includes a bus705and a network interface (NI)712.

Computer710may include or have access to a computing environment that includes one or more user input devices716, one or more output devices718, and one or more communication connections720such as a network interface card or a USB connection. The one or more user input devices716can be a touch screen and a stylus and the like. The one or more output devices718can be a display device of computer, computer monitor, TV screen, plasma display, LCD display, display on a touch screen, display on an electronic tablet, and the like. The computer710may operate in a networked environment using the communication connection720to connect to one or more remote computers. A remote computer may include a personal computer, server, router, network PC, a peer device or other network node, and/or the like. The communication connection may include a Local Area Network (LAN), a Wide Area Network (WAN), and/or other networks.

The memory704may include volatile memory706and non-volatile memory708. A variety of computer-readable media may be stored in and accessed from the memory elements of computer710, such as volatile memory706and non-volatile memory708, removable storage701and non-removable storage714. Computer memory elements can include any suitable memory device(s) for storing data and machine-readable instructions, such as read only memory (ROM), random access memory (RAM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), hard drive, removable media drive for handling compact disks (CDs), digital video disks (DVDs), diskettes, magnetic tape cartridges, memory cards, Memory Sticks™, and the like; chemical storage; biological storage; and other types of data storage.

“Processor” or “processing unit,” as used herein, means any type of computational circuit, such as, but not limited to, a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, explicitly parallel instruction computing (EPIC) microprocessor, a graphics processor, a digital signal processor, or any other type of processor or processing circuit. The term also includes embedded controllers, such as generic or programmable logic devices or arrays, application specific integrated circuits, single-chip computers, smart cards, and the like.

Embodiments of the present subject matter may be implemented in conjunction with program modules, including functions, procedures, data structures, application programs, etc., for performing tasks, or defining abstract data types or low-level hardware contexts.

Machine-readable instructions stored on any of the above-mentioned storage media are executable by the processing unit702of the computer710. For example, a computer program725may include machine-readable instructions capable of triggering handwriting recognition according to the teachings and herein described embodiments of the present subject matter. In one embodiment, the computer program725may be included on a CD-ROM and loaded from the CD-ROM to a hard drive in non-volatile memory708. The machine-readable instructions cause the computer710to encode according to the various embodiments of the present subject matter. As shown, the computer program725includes a handwriting recognition engine730.

In operation, the user input device716draws one or more strokes of a desired sub-word unit. The handwriting recognition engine730coupled to the user input device716that is responsive to the one or more strokes of the desired character determines a first trigger stroke in the drawn one or more strokes that can be used to trigger the sub-word unit recognition upon recognition of the one or more strokes. The handwriting recognition engine730then triggers sub-word unit recognition for the drawn one or more strokes based on the determined first trigger stroke. The handwriting recognition engine730then produces a first candidate sub-word unit upon triggering the sub-word unit. The output device716then displays the produced first candidate sub-word unit. The user input device716then selects the displayed first candidate sub-word unit by touching the displayed first candidate sub-word unit.

The operation of the computer system700for triggering the handwriting recognition is explained in more detail with reference toFIGS. 1-6.

The handwriting recognition engine of the present subject matter is modular and flexible in terms of usage in the form of a “Distributed Configurable Architecture”. As a result, parts of the handwriting recognition engine may be placed at different points of a network, depending on the model chosen. For example, the handwriting recognition engine can be deployed in a server and the input and output data elements streamed over from a client to the server and back, respectively. The handwriting recognition engine can also be placed on each client, with the database management centralized. Such flexibility allows faster deployment to provide a cost effective solution to changing business needs. Further, the above-described technique utilizes certain parts of a sub-word unit that are repetitive, which is natural to a user and does not require learning any scheme, to trigger handwriting recognition. Basically, the above-described technique uses parts of writing to reliably recognize the completion of the writing of the sub-word unit, which can be used to trigger the handwriting recognition. The technique requires the users to write the usually way they are used to write sub-words and does not require them to learn any new techniques and thus provides a better user's experience when drawing a sub-word unit and entering the drawn sub-word unit into a computer.

The above description is intended to be illustrative, and not restrictive. Many other embodiments will be apparent to those skilled in the art. The scope of the subject matter should therefore be determined by the appended claims, along with the full scope of equivalents to which such claims are entitled.

The above-described technique provides various embodiments for triggering handwriting recognition. It is to be understood that the above-description is intended to be illustrative, and not restrictive. Many other embodiments will be apparent to those of skill in the art upon reviewing the above-description. The scope of the subject matter should, therefore, be determined with reference to the following claims, along with the full scope of equivalents to which such claims are entitled.

As shown herein, the present subject matter can be implemented in a number of different embodiments, including various methods, a circuit, an I/O device, a system, and an article comprising a machine-accessible medium having associated instructions.

Other embodiments will be readily apparent to those of ordinary skill in the art. The elements, algorithms, and sequence of operations can all be varied to suit particular requirements. The operations described-above with respect to the method illustrated inFIG. 1can be performed in a different order from those shown and described herein.

FIGS. 1-7are merely representational and are not drawn to scale. Certain proportions thereof may be exaggerated, while others may be minimized.FIGS. 1-7illustrate various embodiments of the subject matter that can be understood and appropriately carried out by those of ordinary skill in the art.