Display of ink input and digital recognition results

One embodiment provides a method, including: accepting, at an input surface, ink input; determining, using a processor, typeset for the ink input; providing, on a display, a combined display of the ink input and the typeset; where the combined display visually associates the ink input and the typeset. Other aspects are described and claimed.

BACKGROUND

Electronic devices such as tablets, smart phones, laptop computers, kiosks, payment stations, etc., increasingly afford users an opportunity to enter handwriting input, hereinafter “ink input.” The ink input is provided by a user's finger or stylus, e.g., to a touch screen or other input surface, and displayed on the screen. The ink input is changed into digitized text or may be converted into a graphic or image (hereinafter “typeset”), e.g., for insertion into another application such as an email or word processing program.

When providing such digital handwriting input to a conventional device, there are some shortcomings. As may be appreciated by use of conventional devices, currently the conversion from ink input to typeset is not perfect, particularly for drawing or other symbol data. Moreover, ink input is typically converted into typeset and thereafter the ink is not retrievable. Thus, fixing a mistake in the conversion is difficult. Sometimes only a minor modification to the ink data is needed to get a correct translation, such as crossing a “t” or extending/shortening a loop on an “e” or like changes. Conventionally, however, there is no way to go back to ink data after conversion (which is usually necessary to share the content with others).

This and other difficulties stem from an approach to ink input that treats the ink input as unidirectional, i.e., from ink input to typeset, and conventional systems do not afford the user the opportunity to see the typeset and ink input at the same time or switch rapidly between the two representations of the user input.

Thus, even if a user would prefer to see the ink input in display form, since that is the form in which the content was collected, and the brain is better able to recall and associate the abstract content with the visual input and memory, and even the muscle memory from the initial writing, there are no known existing solutions to treat the input data as both ink input and typeset simultaneously, and for the user to be able to see the user input data in both forms.

BRIEF SUMMARY

In summary, one aspect provides a method, comprising: accepting, at an input surface, ink input; determining, using a processor, typeset for the ink input; providing, on a display, a combined display of the ink input and the typeset; wherein said combined display visually associates the ink input and the typeset.

Another aspect provides a device, comprising: a display; an input surface; a processor operatively coupled to the display and the input surface; a memory device that stores instructions executable by the processor to: accept, at the input surface, ink input; determine typeset for the ink input; provide, on the display, a combined display of the ink input and the typeset; wherein said combined display visually associates the ink input and the typeset.

A further aspect provides a product, comprising: a storage device having code stored therewith, the code being executable by a processor and comprising: code that accepts, at an input surface, ink input; code that determines, using a processor, typeset for the ink input; code that provides, on a display, a combined display of the ink input and the typeset; wherein said combined display visually associates the ink input and the typeset.

DETAILED DESCRIPTION

As described herein, users face a difficulty in that conventional input technology tends to display ink input only until the user enters it as typeset, but not thereafter. Thus, a user that wants to switch between the ink input display and the typeset display, or view both at the same time, is left frustrated. An existing approach provides a user with the ability to leave the ink input display as an ink representation in order to copy it out as typeset. However, in all known cases, when the original ink input data is converted to typeset, there is no way to convert it back to retrieve the ink input display, save performing an undo action, which is only available briefly.

Also, when copying the ink input data out as typeset, any modifications to that typeset after it has been pasted into a new location do not get fed back into the original source ink input display data, so that when a copy is performed again, the same modifications will need to be performed again. Further, this problem persists in other functionality, e.g., the modifications made to the typeset associated with ink input in conventional systems will not be available for the ink search functionality.

When correcting ink input to typeset text translations, the primary existing solution is to write something as ink input, press a button to transform it into typeset text, and use the undo stack to go back to ink input display if the translation was wrong, or to delete the incorrect word and re-write it in ink input again, and then try to convert it again. Another method is to just fix the converted typeset text with a keyboard and mouse, eschewing the stylus and handwriting input method.

As may be appreciated, the conventional approaches to handling ink input leave much to be desired. Accordingly, an embodiment embraces the dual nature of digital stylus-based input, i.e., the hand-written or drawn ink input and the digitized text or graphic transformation, referred to herein as typeset.

An embodiment provides a side-by-side view similar to a translated book, with the original ink input displayed on one side and the translated typeset displayed on the other side.

An embodiment provides a flashlight view, where one representation is dominant or displayed by default and another view or window is used to show the other representation. An embodiment stores the ink input data and the typeset data such that both may be displayed, simultaneously, and/or the user may switch back and forth between the display views rapidly.

For example, in a flashlight view, the dominant or default representation may be ink input, whereas a view or window will show the other, i.e., the typeset in this example. The position of the view or window may be determined in a variety of ways, e.g., determined by the hover position of the stylus, a touch or hover-touch input event, a mouse or cursor position, etc. The view or window may be displayed in a variety of shapes, e.g., circular, rectangular, etc. The view or window may be represented as a dynamic window, e.g., a floating window displayed on top of the default view or representation. The view or window may be located in a variety of locations, e.g., in-plane with the default view, offset there-from, etc.

The size or bounds of the view or window may be scaled, e.g., to accommodate the larger size of the ink input, or fixed in size. For example, if the flashlight mode is used to display a window over a long word such as antidisestablishmentarianism, then depending on the mode, either the whole word may be displayed into the window or view representation, or the radius (i.e., predetermined amount) may be fixed to the window's anchor position (e.g., hover position of the stylus), with the result that less than or more than the entire ink input word is displayed in a transformed window.

An embodiment may dynamically adjust either or both of the display of the ink input and the display of the typeset. For example, and particularly for a typeset display, there is commonly a large scale difference between ink input and typeset views, particularly if a small typeset font is chosen. As such, an embodiment may adjust, e.g., scale, the display (either or both of the ink input display and the typeset display) such that the scale differences are addressed appropriately.

An embodiment may re-flow the default or dominant view to make room for the window or view of the other representation. This mode may be used primarily for a default typeset view and an in-plane ink input view. In order to have the larger ink input representation fit without overlapping the typeset display, the typeset data may be moved (re-flowed) to accommodate the size of the ink input.

In side-by-side mode, the ink input display is typically larger than the typeset display, and so an embodiment may implement several methods to handle this occurrence. In an embodiment, the inter-line spacing of a paragraph of the typeset display may be changed such that the overall height of the paragraph matches the overall height of the ink display of the same paragraph. In another embodiment, the top of each paragraph (i.e., a paragraph of ink input and a paragraph of associated typeset) may be aligned, leaving the typeset display with the previous or default inter-line spacing so that it appears as it would when exported to another application (e.g., an e-mail application). In another embodiment, the cursor position (which includes any position indicated by a user, whether by moving an on-screen cursor, providing hover input, providing touch input, etc.) in a first representation (e.g., typeset) may be used to highlight data in the other representation (e.g., ink input). That is, when the user has the cursor position in the middle of a word or drawing in the typeset display, the same word or drawing is highlighted in the ink display side (and vice-versa).

With any or all of these methods, the user is able to both keep the ink display of the content on screen and maintain a correct translation of the ink input content to typeset form, thus enabling the ability to search and export the content.

System100typically includes one or more of a WWAN transceiver150and a WLAN transceiver160for connecting to various networks, such as telecommunications networks and wireless Internet devices, e.g., access points. Additionally, devices120are commonly included, e.g., an audio capture device such as a microphone. System100often includes a touch screen170for data input and display/rendering. System100also typically includes various memory devices, for example flash memory180and SDRAM190.

Information handling device circuitry, as for example outlined inFIG. 1orFIG. 2, may be used in devices such as tablets, smart phones, personal computer devices generally, and/or electronic devices that accept ink input and process the ink input to form a digitized form thereof, e.g., typeset. An embodiment facilitates a combined display of the ink input and the typeset (which as used herein includes any graphic/image formed from the ink input), such that the user may view both in a display. As described further herein, the ink input and typeset is logically associated, e.g., by performing a handwriting analysis on the ink input to identify a typeset word or words, and thus the ink input and the typeset may be visually associated in the combined display using the logical association and potentially other parameters, such as user provided input, as further described herein. Specific examples of the combined display as well as example functionality of the various embodiments are described in further detail in connection withFIGS. 3-6.

Referring now toFIG. 3, a device300such as a tablet computing device, etc., provides a combined display301that includes both ink input304and typeset303for the ink input. In the example illustrated inFIG. 3, the user has provided ink input304using a stylus and handwriting recognition software has converted the ink input304into typeset303. The combined display301illustrated by way of example inFIG. 3includes a side-by-side view, where the ink input304and the typeset303are visually associated. As described herein, the visual association may take a variety of forms.

It may first be observed that an embodiment provides for the ink input304to persist while the typeset303is also displayed. This allows a user to review the interpretation of the handwritten ink input304, i.e., the typeset303chosen by the device's handwriting recognition software. Here, the ink input304and the typeset303are further visually associated in that the first line of ink input304and the first line of typeset303are aligned with one another in the combined display301. As described herein,FIG. 3illustrates how the typeset303often takes up less space than the ink input304.

Moreover, illustrated inFIG. 3is another visual association between the ink input304and the typeset303. As shown, as the user hovers the stylus tip302over a word in the typeset303, in this example “galleries,” the typeset display303of the word “galleries” is visually distinguished (e.g., highlighted), as is the display of the corresponding data of the ink input304. Thus, the user can quickly link or associate the typeset303with a particular part of the handwritten ink input304displayed in the combined display301. An embodiment may also highlight typeset303in response to a user hovering a tip of a stylus302(or finger, etc.) over part of the ink input304. This facilitates quicker review of the ink input304and revision thereof.

Another type of combined display401is illustrated inFIG. 4. Here, the combined display includes a dominant or default representation of typeset403; however, this is a non-limiting example and the dominant or default representation may be that of ink input, e.g., as illustrated in combined display301at304ofFIG. 3. The combined display dynamically adjusts such that if the user hovers a tip of the stylus402over a particular part of the typeset, here the word “coordinate,” indicated at404, the ink input data that the user provided to generate that typeset representation is displayed in the combined display. This facilitates a user's ability to correct errant typeset representations, e.g., if the device400makes a mistake in the handwriting input analysis, a user is quickly apprised of what the original ink input looked like.

FIG. 5shows another example of a combined display501. Here, the user has moved the tip of the stylus502over the word “with,” as indicated at504, and the handwritten ink input used to generate that typeset result is displayed in a window in the same plane as the typeset503. The window, as with other views, may be an overlay or the underlying display data (here typeset503) may be reflowed and the other data (here ink input504) may be inserted, e.g., as a graphic.

Thus, the user may again be quickly apprised of what handwritten ink input data was used to generate the typeset result. As with the other embodiments described here, the dominant or default representation, here that of typeset503, may be reversed, i.e., ink input may be the default or dominant representation in the combined display501.

An embodiment therefore facilitates the visual association of typeset representations of ink input data. The visual association is made possible by the simultaneous display capability where ink input and typeset are provided in a combined display.

As shown inFIG. 6, an embodiment accepts input at601and performs a handwriting analysis to determine a typeset for the ink input. If a typeset determination may be made, e.g., identifying a word, phrase or symbol, as illustrated at602, an embodiment will provide a combined display as shown at603. A combined display may be a side-by-side display, as illustrated inFIG. 5, or a dynamic combined display that displays alternate versions (of ink input or typeset, as dictated by the default or dominate mode) in response to user input. Each type of combined display is facilitated by storing the ink input data and the typeset data for use in various forms of combined display.

If a determination cannot be made at602as to what typeset representation is appropriate for the ink input data, an embodiment may take no action and await further input, an embodiment may take a predetermined action, such as providing a suggestion that the ink input be inserted as a graphic, etc.

The combined display provided at603may be adjusted in a variety of ways. For example, as discussed, a visual association between the typeset and the ink input may be provided, as indicated at604. The visual association may include aligning the handwritten ink input display with the typeset display, aligning the typeset display with the handwritten ink input display, or a combination of the foregoing. Moreover, the visual association may include scaling, resizing and/or re-flowing the typeset, the handwritten ink input, or a suitable combination of the foregoing. This permits the visual display of the typeset and the ink input to be adjusted according to predetermined settings, user preferences (implied or expressed), etc.

In a further example, as illustrated inFIG. 4andFIG. 5, the combined display and thus the visual association may be dynamically implemented, e.g., in response to a cursor position, such as the location of the stylus tip. Thus, an embodiment visually associates typeset (e.g., a word) with ink input (e.g., the handwritten ink input for that word). The visual association may be implemented by displaying a typeset representation of ink input, displaying an ink input representation of typeset, and/or highlighting (or otherwise visually indicating) one or more of the typeset and the ink input associated with a cursor location. As described herein, the nature of the visual association may take a variety of forms, e.g., highlighting an entire word, an entire line of text, a predetermined portion of data (e.g., part of a word within a defined radius of the cursor location/stylus tip), etc.