Formatting handwritten content

One embodiment provides a method, including: accepting, in an input and display device, handwriting ink strokes; providing, using a handwriting recognition engine, one or more machine words for the one or more ink strokes; determining a characteristic of the one or more handwriting ink strokes; changing, based on the characteristic of the handwriting ink strokes, input formatting for the one or more machine words; and inputting the one or more machine words according to the input formatting. Other embodiments are described and claimed.

BACKGROUND

Information handling devices (“devices”), for example cell phones, smart phones, tablet devices, laptop computers, and the like permit users to input handwriting using a finger or pen/stylus. This allows users to write more naturally and without the use of a keyboard.

Conventionally a handwriting field, box or pane is presented to the user as an overlay, e.g., a rectangle in a lower portion of a touch screen display. In this area the user may provide input handwriting strokes, e.g., letters, numbers, characters, symbols, etc. The device employs software that uses the input handwriting strokes, which are generally presented on screen to provide visual feedback to the user, as input by converting the handwriting stroke input locations on the touch screen into machine text according to the language setting of the device. Alternatively, a graphic or image of handwriting input may be placed into an underlying application, e.g., an image of the user's signature.

BRIEF SUMMARY

In summary, one aspect provides a method, comprising: accepting, in an input and display device, handwriting ink strokes; providing, using a handwriting recognition engine, one or more machine words for the one or more ink strokes; determining a characteristic of the one or more handwriting ink strokes; changing, based on the characteristic of the handwriting ink strokes, input formatting for the one or more machine words; and inputting the one or more machine words according to the input formatting.

Another aspect provides an apparatus, comprising: an input and display device; a processor operatively coupled to the input and display device; and a memory device that stores instructions executable by the processor to: accept, in the input and display device, handwriting ink strokes; provide, using a handwriting recognition engine, one or more machine words for the one or more ink strokes; determine a characteristic of the one or more handwriting ink strokes; change, based on the characteristic of the handwriting ink strokes, input formatting for the one or more machine words; and input the one or more machine words according to the input formatting.

A further aspect provides a product, comprising: a storage device having code stored therewith, the code being executable by a processor of an electronic device and comprising: code that accepts, in an input and display device, handwriting ink strokes; code that provides, using a handwriting recognition engine, one or more machine words for the one or more ink strokes; code that determines a characteristic of the one or more handwriting ink strokes; code that changes, based on the characteristic of the handwriting ink strokes, input formatting for the one or more machine words; and code that inputs the one or more machine words according to the input formatting.

DETAILED DESCRIPTION

Properly converting the structure of a handwritten document to machine form input (machine text, machine words, or herein simply referred to as “typeset”) is difficult. Paragraphs, lists, tables, and other structural elements are created visually by their division in to blocks and the relative positioning of those blocks on the page. There are few unequivocal indicators in the handwriting script of the user's intent.

Input method editors (IMEs) and applications that natively support handwriting inputs do not attempt to analyze or understand the document structure of the handwritten content and simply record the handwriting as it was written, e.g., in the original graphical form. This is the default behavior for EVERNOTE and ONENOTE applications. For the purpose of searching, the handwriting may be converted by such applications to machine text, as a background task and out of sight of the user, where understanding the document structure is not so important. EVERNOTE is a registered trademark of Evernote Corp. in the United States. ONENOTE is a registered trademark of Microsoft Corp. in the United States.

Other approaches convert handwriting immediately to typeset text as it is written and position the typeset text by normal word processing rules. New typeset text is added to the document at the position of the typeset cursor in the underlying application, rather than at the place it is written. Such repositioning of the handwriting into converted typeset according to normal word processing forms is how IMEs and handwriting “keyboards” typically function. In another example, the typeset text is added to the document in a position approximating where it was originally handwritten, but no structure beyond a letter or a word is comprehended. This behavior is seen in the ONENOTE application's convert to text function and in the MYSCRIPT Notes Mobile application. MYSCRIPT is a registered trademark of Myscript Societe Anonyme France in the United States.

A conventional approach to handling handwriting input may also present a static template with lines or other guides that direct the user towards creating a specific document structure while inputting handwritten content. Here, the handwriting is assumed to align with and be structured by the lines, rather than written free-form. Any given stroke must belong to a single line of text, and the lines are consistent across the whole page. Writing on successive lines, without intervening white space, is assumed to be part of the same paragraph. This model is seen in the Stylus Labs WRITE application. WRITE is an unregistered name used in connection with a word processing application for handwriting by Stylus Labs in the United States.

Another conventional approach is to record the handwritten strokes verbatim and thereafter analyze the handwriting content and document structure offline. This model is seen in the ABBYY, OCRFeeder, and OCRopus applications. The analysis and blocking is typically done as a preprocessing step before text recognition. It is an outgrowth of OCR for typeset text documents and as such leans heavily on finding clear-cut graphical boundaries and spacing between blocks. Handwritten documents often do not satisfy this assumption. ABBYY is a registered trademark of Abbyy Software Ltd. Corp. in the United States. OCRFeeder is an unregistered name used in connection with a document layout analysis and optical character (OCR) recognition system by the GNOME Project in the United States. OCRopus is an unregistered name referring to a Python-based OCR package using recurrent neural networks available via GitHub, Inc. in the United States.

An embodiment employs a different approach as compared to conventional approaches in that it analyzes a document of handwritten text as it is being written and displays to the user the interpreted document structure. This permits both a visual display of how the handwritten ink strokes are being interpreted and may also be used to gently guide the writer to extend or correct the structure going forward.

A handwritten document's structure may be discovered by analyzing the blocks of handwriting and their relative positioning while the document is being written. Subtle feedback mechanisms indicate to the writer the derived structure, making it easy to continue within the same structure or correct where it has been misinterpreted.

By way of example, for line spacing, a handwriting recognition engine discerns lines of handwritten text. By allowing the writer to write 2-3 lines of text free-form, then discerning lines of text in the provided ink input, the writer's character height and line spacing can be derived.

Once the line spacing is known for a particular user's handwritten strokes, automatic and disappearing guidelines, displayed for example as faint notebook-style horizontal lines, may be provided by an embodiment to encourage the writer to continue with the same spacing. The guidelines may be shown only in the locality of the pen tip, and may be removed from view when the pen is withdrawn from the screen for an extended period.

After a few lines of handwriting input is provided, e.g., the first 2-3 lines have been written, an embodiment may detect where the user is placing the left and right margins of the handwritten input. An embodiment may therefore detect the line length of the handwritten input and this can be used to define a text block's side boundaries or margins.

Typeset text conversion and wrapping also may be provided by an embodiment. For example, with the top, bottom, left, and right boundaries or margins of a handwriting block known, e.g., corresponding to a paragraph of handwritten text input, the block can be considered a paragraph and converted to typeset text form. The converted text may be aligned with the original handwritten block periphery and wrapped within its side-to-side extents. Thus, an embodiment honors the original intent of the writer by associating typeset text blocks with the detected handwritten block boundaries.

A difficulty solved by an embodiment is whether to include new handwriting ink stroke within an existing block or a new block should be created for it. If the new handwriting falls within the extent of an existing block, the new handwriting is clearly meant to be added to that block. If the new handwriting is written in white-space far from any existing block, e.g., beyond a predetermined distance, a new block may be created. Between these two extremes, an embodiment defines a rule where, if the new handwriting falls within one line-spacing below the current block, or within 1-2 em-widths to the right, the new handwriting is to be added to the existing block. It will be appreciated that such hot-zone or buffer area widths are examples and may be adjusted, either via user configuration or dynamically.

As an aid to understanding this block determining behavior, the hot-zone or buffer area(s) may be visually distinguished, e.g., highlighted with shaded semi-transparent underlay boxes, hatching, or other graphical marks. Alternately, a highlighting may be placed behind the whole area of the existing block (e.g., paragraph block) plus the hot-zone(s). As with disappearing guidelines, the highlighting or other visual indication may be exposed only when the user has the pen near or on the writing surface, only when the user approaches a hot-zone with ink strokes, etc.

An embodiment may automatically format handwritten lists and outlines as well. By way of example, single and multi-level lists may be detected by a consistent indentation of the left margin relative to a preceding block and/or by the writing of a “bullet” type list symbol (e.g., dot, dash, circle, box, check-mark, arrow, etc.) at the start of the list block's first line. Once the first item of a list has been created, horizontal and vertical disappearing guidelines may be shown to the nudge or guide the writer towards continuing the alignment and spacing he or she has started. If further indentation is detected within a list, preference may be given to interpreting the new block as the start of a new sub-list nested within the current list.

To automatically format a table, an embodiment may use an explicit input such as a dedicated table gesture input. For example, because it may not be apparent that the handwritten ink strokes represent a table until a user has written several blocks, table creation may begin with an explicit gesture such as drawing a box or large “T” shape with a pen or finger tip. With a “T” shape gesture, the position of the vertical division between the initial two columns may be set by the “T” gesture's vertical stroke. Once the table framing has been made, each table cell may behave as a paragraph block. Text wraps within the block's boundaries and the column or row grows as the user writes into a hot-zone, e.g., at the bottom and right sides of the existing block.

Table editing may be supported via explicit gestures. For example, dedicated gestures may be provided, using a pen or finger tip, which are used for modifying the structure or content of a table. By way of example, a circle around the outside border of a row or column may select the whole column. As another example, slashing through the same boundary with an ink stroke may cause the row or column to be deleted. A caret-arrow ({circumflex over ( )}) written with the point touching a table boundary, outside or inside the table, may cause a new row or column to be inserted in the direction the arrow points.

System100typically includes one or more of a WWAN transceiver150and a WLAN transceiver160for connecting to various networks, such as telecommunications networks and wireless Internet devices, e.g., access points. Additional devices120are commonly included, e.g., a camera, short range wireless devices, a microphone, etc. System100often includes a touch screen170for data input and display/rendering. System100also typically includes various memory devices, for example flash memory180and SDRAM190.

Information handling device circuitry, as for example outlined inFIG. 1orFIG. 2, may be used in devices such as tablets, smart phones, personal computer devices generally, and/or other electronic devices. Users may employ these devices for various functions, e.g., inputting handwriting strokes as an input modality to an Internet search application, an email application, a text messaging or instant messaging application, with fillable forms, etc. For example, the circuitry outlined inFIG. 1may be implemented in a tablet or smart phone embodiment, whereas the circuitry outlined inFIG. 2may be implemented in a laptop personal computer embodiment. In each case, the device may include an input and display device, e.g., a touch screen or the like, which acts to accept handwriting strokes and provides visual displays of input characters, conversions of handwritten characters or strokes to machine text, presentation of candidate handwriting recognition words and the like.

Devices including circuitry such as that outlined inFIG. 1and/orFIG. 2(or combinations thereof) may implement an overlay input application, as described herein, which formats typeset text according to the user's handwriting ink strokes. In this regard,FIG. 3illustrates a general overview of an example overlay input application.

As illustrated inFIG. 3, a user may provide typeset input302to an underlying application (which may include an end user application or an operating system (OS))301, in this non-limiting example an email application, by way of handwriting input303. An overlay input application acts to provide an input overlay field that directly overlays the underlying application's301input field. The use of an overlay application permits system wide handwriting formatting as described herein, i.e., across multiple applications. However, it should be noted that an overlay application is a non-limiting example implementation used descriptively throughout, and the various embodiments may be provided within the application itself rather than including the functionality within an overlay application.

In the illustrated example ofFIG. 3, the overlay input field is provided on top of the email main message body field, as this is where the user is currently writing.

The user's handwriting ink strokes303, currently appearing as “get the hotdogs and hamburgers” is subjected to an OCR like handwriting recognition processing by a handwriting recognition engine. The overlay input application provides a preview window304that, among other things, provides the user with a visual feedback of the typeset words305identified from the handwriting ink strokes303.

The handwriting ink strokes303may thus be placed or input into the underlying input field of the email application301, as indicated for typeset302. The formatting (e.g., location, logical layout, etc.) of the input typeset305derived from the handwriting ink strokes303is the subject of an embodiment.

InFIG. 4, by way of illustrative example, the user has previously provided handwriting ink strokes. The overlay input application has converted these to machine input, along with a formatting, and provided the same to the underlying application to display typeset402. As may be seen, the typeset “Dear Friends,” is off-set from (visually above) the paragraph of text beginning “If you happen to . . . ” inFIG. 4. To provide a user with improved formatting of such typeset402, an embodiment may provide guidelines (405a,405b, and407) based on detected handwriting ink stroke characteristics.

In the non-limiting example ofFIG. 4the user's prior ink strokes (not explicitly illustrated) included a lateral or horizontal characteristic, i.e., they began and ended at approximately the location of guidelines405aand405b, respectively. Accordingly, an embodiment has input the typeset402in the underlying application's input field at these locations. In other words, the user's handwriting ink strokes were used to set the left and right block boundaries or margins for the paragraph of typeset input.

Likewise, the user's handwriting ink strokes included a line spacing that permitted an embodiment to calculate that a next line should be positioned just above guideline407. This may be determined, for example, using the previous 2-3 lines of handwriting ink strokes input by the user. Similarly, an embodiment may calculate or determine a hot-zone or buffer area, e.g., denoted by406, within which the user's next handwriting ink stroke should begin if it is to be considered part of the existing block (i.e., the one wrapping the paragraph of typeset beginning with “If you happen . . . ”). As described herein, this buffer area or hot-zone may be visually presented to the user to guide the user in providing input to be included or excluded with an initially set block.

As shown inFIG. 4, the user has input new handwriting ink strokes403outside of (below) the guideline407, rather than above, and additionally has placed the handwriting ink strokes403outside of buffer or hot-zone (406-407). This positioning of the new handwriting ink strokes “Hope to see . . . ” may be used to determine a characteristic associated therewith, i.e., the user has not maintained the previous line spacing but has chosen to input the ink strokes beyond the calculated buffer area406. This indicates that the user desires to a new block to be created, i.e., for inputting the typeset into a new paragraph.

Turning to the example illustrated inFIG. 5(A-B), an embodiment may adapt to the user's handwriting ink strokes dynamically in many ways. As illustrated inFIG. 5A, the user has provided ink strokes “We Need” and an embodiment has provided guidelines506and508in response thereto. This indicates to the user that if further ink strokes are provided between guidelines506and508, these ink strokes will be associated with the prior ink strokes, i.e., “We Need.”

As shown, the user has decided to provide handwriting ink strokes of “-tea” on the overlay input field below guideline508. Normally this may indicate that the user wishes to begin a new line of text. Had the user provided the new ink strokes farther below line508, e.g., outside of a buffer area or hot-zone (not explicitly illustrated inFIG. 5A), this may be characterized by an embodiment as a formatting change wherein a new block is created for the user.

The user has also provided two characteristics with the handwriting ink strokes “-tea” that may be used (alone or in combination) to implement a formatting change of starting a list. The first characteristic is the location of the beginning of the ink strokes, i.e., indented to the right of left block boundary505. Here it may be appreciated that the indentation may be detected and trigger the beginning of a list.

Moreover, the user has provided an explicit gesture input of “-” at509that may be used itself to trigger a bulleted list format change. As such, referring toFIG. 5B, an embodiment may implement list formatting such that the typeset conversion of “We Need” and “-tea” are provided as a list. As shown inFIG. 5B, an embodiment handles formatting of “We Need” ink strokes and “-tea” ink strokes. Here, the typeset for “We Need” ink strokes has been located at left most guideline505, corresponding to the user's ink strokes of “We Need” aligning with that block boundary. However, the typeset for “-tea” is indented for the user in a list form. This permits the user to have his or her handwriting ink stroke formatting honored in the underlying application.

As further illustrated inFIG. 5B, additional guidelines510may be provided, i.e., to guide the user in providing further list inputs. Should the user wish to enter a new paragraph, e.g., below the list, he or she need only start the new handwriting ink strokes near boundary block boundary505to create a new block of text.

FIG. 6(A-B) illustrates another example of using a gesture input to format handwriting ink strokes. As illustrated inFIG. 6A, the handwriting ink strokes609include the characteristic of a predetermined pen gesture, i.e., a “T” shape as described herein. When the user provides this gesture609(or other predetermined or configured, dedicated gesture) an embodiment may implement an associated formatting change. By way of example, as illustrated inFIG. 6B, the “T” shaped gesture may be associated with a table format change. Thus, an embodiment may switch from a normal paragraph text block format (and inputting a typeset letter T) to a table format, e.g., including paragraph blocks in each cell of the table615for text input.

As described throughout, additional gestures may be provided. By way of another example, a user may input a caret or insert gesture, e.g., as illustrated at709ofFIG. 7A, such that an associated function is implemented. Here, an embodiment has been configured to insert new cells in the table715ain response to detecting the gesture input709. Thus, table715amay be changed to table715b(FIG. 7B) in response to detecting handwriting ink stroke709. As has been described, other gestures may be used to implement formatting changes. These gestures may be used to format an existing format structure (e.g., add cells to a table, add items to a list, delete items there-from using a strike through gesture, highlight items, etc.) and/or may be used to format or change substantive content (e.g., delete text, insert text, highlight text, etc.).

Accordingly, and referring generally toFIG. 8, an embodiment permits a user to format underlying typeset in-line using handwriting ink strokes. An embodiment analyses characteristic(s) of the ink strokes in order to make determinations as to formatting changes and also provides the user with visual cues regarding what formatting decisions have been made.

As shown inFIG. 8, handwriting ink strokes are detected at801. These may take the form of letters, numbers, characters, symbols, gestures, a graphic and the like. If typeset words may be identified by a handwriting recognition engine at802, these may be presented to the user, e.g., in a preview window (illustrated by way of example inFIG. 3).

An embodiment uses the handwriting ink strokes to determine characteristics thereof at803. As described herein, this may take the form of determining height or width of individual characters or words, spacing there-between, line spacing characteristics, boundary or margin characteristics, detection of a dedicated gesture input, a combination of the foregoing, etc. If a characteristic is determined at804, an embodiment may use this characteristic to implement a formatting change at805. By way of non-limiting example, a user providing handwriting input including an indent or special bullet gesture may be used to implement a change from paragraph block formatting to list formatting, as illustrated in the example ofFIG. 5(A-B). This permits an embodiment to input the typeset to the underlying application at806.

If characteristic(s) are not determined at804, e.g., the user has simply continued within the same block, continues the same line spacing, has not input a special gesture, etc., an embodiment may maintain a static or default formatting at807, e.g., continuing to consider the handwriting ink strokes as input into an existing block. Thus, the typeset may be input to the underlying application using the existing formatting at808. It should be noted, however, that the use of the existing formatting may include some formatting changes. For example, an embodiment may clean-up the handwriting ink strokes' formatting, such as by aligning the line spacing and/or an embodiment may make wholesale changes, such as re-wrapping a paragraph when a user adds new text in the middle or changes the block width, e.g., via moving a block boundary or re-defining the margin settings.

An embodiment therefore represents a technical improvement in formatting handwriting ink strokes for typeset input in an underlying application. This assists the user by more intelligently linking the formatting of the input to the context of handwriting or ink input itself. The various embodiments produce a formatting, including indications thereof, that more closely match the user's intent by taking into consideration the actual characteristics of the ink strokes in determining a format for inputting the typeset.

It should be noted that the various functions described herein may be implemented using instructions that are executed by a processor and stored on a device readable storage medium, referred to herein as a non-signal storage device. A non-signal storage device may be, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a storage medium would include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a non-signal storage device is not a signal, is non-transitory, and “non-transitory” includes all media except signal media.