System and method of handwriting recognition in diagrams

A system, method and computer program product for hand-drawing diagrams including text and non-text elements on a computing device are provided. The computing device has a processor and a non-transitory computer readable medium for detecting and recognizing hand-drawing diagram element input under control of the processor. Display of input diagram elements in interactive digital ink is performed on a display device associated with the computing device. One or more of the diagram elements are associated with one or more other of the diagram elements in accordance with a class and type of each diagram element. The diagram elements are re-displayed based on one or more interactions with the digital ink received and in accordance with the one or more associations.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to European Application No. 15290271.4 filed on Oct. 19, 2015, the entire contents of which is incorporated by reference herein.

TECHNICAL FIELD

The present invention relates generally to the field of computing device interfaces capable of recognizing user input handwriting of various graphics and text. In particular, the present invention provides systems and methods for the detection and recognition of input handwritten diagram elements to produce digital diagram documents.

BACKGROUND

Computing devices continue to become more ubiquitous to daily life. They take the form of computer desktops, laptop computers, tablet computers, hybrid computers (2-in-1s), e-book readers, mobile phones, smartphones, wearable computers (including smartwatches, smart glasses/headsets), global positioning system (GPS) units, enterprise digital assistants (EDAs), personal digital assistants (PDAs), game consoles, and the like. Further, computing devices are being incorporated into vehicles and equipment, such as cars, trucks, farm equipment, manufacturing equipment, building environment control (e.g., lighting, HVAC), and home and commercial appliances.

Computing devices generally consist of at least one processing element, such as a central processing unit (CPU), some form of memory, and input and output devices. The variety of computing devices and their subsequent uses necessitate a variety of interfaces and input devices. One such input device is a touch sensitive surface such as a touch screen or touch pad wherein user input is received through contact between the user's finger or an instrument such as a pen or stylus and the touch sensitive surface. Another input device is an input surface that senses gestures made by a user above the input surface. A further input device is a position detection system which detects the relative position of either touch or non-touch interactions with a non-touch physical or virtual surface. Any of these methods of input can be used generally for the handwritten or hand-drawn input of drawings and text which input is interpreted using a handwriting recognition system or method.

One application of handwriting recognition in computing devices is in the creation of diagrams which are hand-drawn on a computing device to be converted into typeset versions. Diagrams are drawings that explain or show arrangement and relations (as of parts). Diagrams generally include shapes having arbitrary or specific meanings and text with relationships to these shapes. There are many type of diagrams, such as flowcharts, organizational charts, concept maps, spider maps, block/architecture diagrams, mind-maps, block diagrams, Venn diagrams and pyramids. Depictions of some typeset and handwritten examples of possible diagrams are illustrated inFIGS. 1 to 6.

FIGS. 1A and 1Brespectively show typeset and handwritten example concept maps10variously having shapes, defining diagram blocks or containers12and connectors14, of different type (e.g., straight arrows, curved arrows), which connect or designate relationships between the diagram blocks12. Further, inFIG. 1Bthe containers12contain text16. Generally in concept maps the connections between the blocks define conceptually related or dependent elements or themes defined by the text in those blocks. The relationship between blocks may be made precise using labels on connectors. The blocks themselves may not be present in the concept map and instead the text (e.g., defined in text blocks having no associated shape or container) may be connected by the connectors.

FIGS. 2A and 2Brespectively show typeset and handwritten example mind-maps20variously having shapes defining diagram blocks or containers12, connectors14, of different type (e.g., straight lines, curved lines), which connect or designate relationships between the diagram blocks12and paths18to certain features or states of the mind maps. Further, inFIG. 2Bthe containers12and paths18have associated text16. Generally in mind-maps the connections between the blocks define possible alternative states or linked ideas from central elements or themes defined by the text in those blocks, and the paths define key features of each alternative state defined by the text on those paths. The blocks themselves may not be present in the mind map and instead the text (e.g., defined in text blocks having no associated shape or container) may be connected by the connectors and paths.

FIGS. 3A and 3Brespectively show typeset and handwritten example flow charts or diagrams30variously having shapes, defining diagram blocks or containers12, of different type (e.g., ovals, rectangles, diamonds), and connectors14, of different type (e.g., straight arrows, bent arrows, branched lines), which connect or designate relationships between the diagram blocks12. Further, inFIG. 3Bthe containers12contain text16; text may also be associated with the connectors. Generally in flow charts the connections between the blocks define procedurally related or dependent elements or steps defined by the text in those blocks. The blocks themselves may not be present in the flow chart and instead the text (e.g., defined in text blocks having no associated shape or container) may be connected by the connectors.

FIGS. 4A and 4Brespectively show typeset and handwritten example organizational charts or tree diagrams40variously having shapes, defining diagram blocks or containers12, and connectors14, of different type (e.g., straight lines, bent lines, branched lines), which connect or designate relationships between the diagram blocks12. Further, inFIG. 4Bthe containers12contain text16. Generally in organizational charts the connections between the blocks define hierarchical relationships of members or functions of an organization or group defined by the text in those blocks. The blocks themselves may not be present in the organizational chart and instead the text (e.g., defined in text blocks having no associated shape or container) may be connected by the connectors.

FIGS. 5A and 5Brespectively show typeset and handwritten example block/architecture drawings50variously having shapes, defining diagram blocks or containers12, having nested relationships (e.g., containers12within other containers12), and connectors14which connect or designate relationships between the diagram blocks12, including between nested blocks. Further, inFIG. 5Bthe containers12and connectors have associated text16. Generally in architecture drawings the nested blocks define arrangement or possession of device or process components, and the connections between the blocks define functional relationships between the blocks defined by the text in those blocks.

FIGS. 6A and 6Brespectively show typeset and handwritten example spider maps60variously having shapes, defining diagram blocks or containers12and connectors14which connect or designate relationships between the diagram blocks12. Further, inFIG. 6Bthe containers12and connectors have associated text16. Generally in spider maps the connections between the blocks and/or text define dependent relationships or states from a central element or theme defined by the text.

The diagrams illustrated inFIGS. 1 to 6are merely examples and other or different elements than those depicted for each diagram type, or different types or forms of the depicted elements themselves, may be present in the diagrams in addition or in the alternative. Further, other definitions of these diagram types are possible as well as combinations thereof. These myriad possible variations of combining the base components of shapes (connections with or without containers) and text in diagrams can cause issues for the accurate recognition of these elements input as hand-drawn or written content to a computing device. Diagrams are particularly used in education and business settings where the user of the computing device creates a diagram, for example, during a lecture or meeting to capture concepts, issues or solutions being discussed. Another frequent use of diagrams is the creation of presentations or reference documentation. Input can be achieved by the user launching a handwritten diagram or sketch application on the computing device which accepts and interprets, either locally in the device or remotely via a communications link of the device, hand-drawn input on a touch sensitive surface or a surface monitored by a relative position detection system.

Conventionally such handwritten diagramming applications are limited in their capabilities to handle the above-described complexity of diagramming and typically constrain users to adopt behaviors or accept compromises which do not reflect the user's original intent. As a result some conventional handwritten diagramming applications force users to navigate menus to select and draw shapes and insert text in relation to shapes. As such, users are unable to draw shapes and connectors naturally or freely. Some other conventional handwritten diagramming applications rely on the order in which users draw different strokes to thereby guide the interpretation for recognition as expected behavior is followed. For example, the user may need to first draw two blocks/boxes before being able to define a connector between those boxes, or may have to draw a box before adding text thereto. This however is difficult for users, as they need to learn and implement the drawing/writing orders required which may need to be re-learnt if the application is not often used, and is non-intuitive, such that the ability to quickly capture diagrams is not supported. For example, a user may wish to prepare presentations on the go with a portable computing device, such as a tablet, or a user may wish to jot down a flow chart that their teacher has drawn in class on a computing device, such as a laptop with a touchscreen, and as such users need to be able to draw clear diagrams with mixed content without being an expert of the dedicated, cumbersome software.

Making the handwritten diagramming application smarter helps support users. That is, the application may be able to distinguish between different shapes, such as between blocks and connectors, and between shapes and text, thereby providing users with more freedom when creating diagrams. For example, U.S. Pat. No. 7,352,902 describes differentiating writing from drawing in input ink by using an ink parser which executes word grouping, writing/drawing classification and drawing grouping. The word grouping is described as being performed by grouping stokes into hierarchies of words, lines and blocks. However, as described in the patent, this grouping of strokes into words during such classification leads to the word groups including non-text strokes, which results in erroneous text recognition when the word groups are sent to a text recognizer, for example. After word grouping, the patent describes that writing/drawing classification is performed. This process is described as including consideration of word, spatial and temporal context features; however these features are mapped to a fuzzy function for classification which means that absolute decisions on whether strokes belong to text or drawings are made at this point, which can lead to misclassification. After classification, the patent describes that drawing grouping is performed by a chart detector to group the drawing stokes into independent objects based on spatial relationships. However, the accuracy of this grouping is influenced by the classification result, such that incorrect objects may be formed.

Even in conventional applications in which hand-drawn shapes and handwritten text are recognized well with reasonable creative freedom offered to users, typically the ability to change the drawn diagrams, such as to edit elements of the diagram to add, omit or replace elements, adapt the diagram to an evolving concept, convert the type of diagram, etc., is limited where only certain operations are available and only available on the typeset version of the diagram, especially with respect to manipulations of the relative positions of diagram elements while retaining recognized relationships, such as connected containers, for example, and not on the handwritten input, so-called digital ink, and/or requires gestures to be learnt or selection to be made via menus, as described above. For example, U.S. Pat. No. 8,014,607 describes an inferred mode protocol which allows certain editing operations to be directly performed on the digital ink. However, the described operations are very limited. Further, no solution is provided for the ability to manipulate the relative position of the diagram elements in digital ink while retaining recognized relationships.

U.S. Pat. No. 7,394,935 describes relative manipulations on the digital ink with respect to resizing and repositioning operations. However, in these operations the digital ink is either merely scaled in accordance with the manipulation, and as such the user would be required to perform further interaction to return the digital ink to its originally drawn dimensions, e.g., moving a container away from its connected container(s) would cause the connector to stretch in both x and y dimensions, or the connectors are ‘reflowed’ by re-computing a backbone (horizontal and vertical lines) that approximates the digital ink of the connector when the connector is resized or changed to a different form (e.g., straight to bent). This requires regeneration of the digital ink, which may be done through normalization of the connector ink through segmentation at high curvature points (cusps) as described in the related U.S. Pat. No. 7,324,691. Accordingly, the resultant manipulated digital ink may be quite different to the originally drawn ink, requiring intervention by users.

SUMMARY

The examples of the present invention that are described herein below provide systems, methods and a computer program product for use in diagram creation with handwriting input to a computing device. The computer program product has a non-transitory computer readable medium with a computer readable program code embodied therein adapted to be executed to implement the method.

The computing device is connected to an input device in the form of an input surface. A user is able to provide input by applying pressure to or gesturing above the input surface using either his or her finger or an instrument such as a stylus or pen. The present system and method monitors the input strokes.

The computing device has a processor and at least one application for detecting and recognizing the handwriting input under control of the processor. The at least one system application is configured to cause display of a plurality of input diagram elements in interactive digital ink on a display device associated with the computing device, associate one or more of the diagram elements with one or more other of the diagram elements in accordance with a class and type of each diagram element, and cause re-display of the diagram elements based on one or more interactions with the digital ink received and in accordance with the one or more associations.

Another aspect of the disclosed system and method provides identification of the class of each diagram element by classifying strokes of the hand-drawn input. The strokes may be grouped based on spatial and temporal information of the input, and the strokes may be classified by building and testing element type probability hypotheses for the groups of strokes. The identified types include text and non-text.

Another aspect of the disclosed system and method provides parsing the classified strokes to a handwriting recognition system for recognition of the diagram elements of the classified strokes.

Another aspect of the disclosed system and method provides identifying the type of each diagram element based on the recognized diagram elements and positional relationships between the diagram elements.

DETAILED DESCRIPTION

Reference to and discussion of directional features such as up, down, above, below, lowest, highest, horizontal, vertical, etc., are made with respect to the Cartesian coordinate system as applied to the input surface on which the input to be recognized is made. Further, terms such as left and right are made in relation to the reader's frame of reference when viewing the drawings. Furthermore, the use of the term ‘text’ in the present description is understood as encompassing all alphanumeric characters, and strings thereof, in any written language and common place non-alphanumeric characters, e.g., symbols, used in written text. Further still, the term ‘non-text’ in the present description is understood as encompassing freeform handwritten or hand-drawn content and rendered text and image data, as well as non-alphanumeric characters, and strings thereof, and alphanumeric characters, and strings thereof, which are used in non-text contexts. Furthermore, the examples shown in these drawings are in a left-to-right written language context, and therefore any reference to positions can be adapted for written languages having different directional formats.

The various technologies described herein generally relate to capture, processing and management of hand-drawn and handwritten content on portable and non-portable computing devices in a manner which retains the inputted style of the content while allowing conversion to a faithful typeset or beautified version of that content. The systems and methods described herein may utilize recognition of users' natural writing and drawing styles input to a computing device via an input surface, such as a touch sensitive screen, connected to, or of, the computing device or via an input device, such as a digital pen or mouse, connected to the computing device or via a physical or virtual surface monitored by a position detection system. Whilst the various examples are described with respect to recognition of handwriting input using so-called online recognition techniques, it is understood that application is possible to other forms of input for recognition, such as offline recognition in which images rather than digital ink are recognized. The terms hand-drawing and handwriting are used interchangeably herein to define the creation of digital content by users through use of their hands either directly onto a digital or digitally connected medium or via an input tool, such as a hand-held stylus. The term “hand” is used herein to provide concise description of the input techniques, however the use of other parts of a users' body for similar input is included in this definition, such as foot, mouth and eye.

FIG. 7shows a block diagram of an example computing device100. The computing device may be a computer desktop, laptop computer, tablet computer, e-book reader, mobile phone, smartphone, wearable computer, digital watch, interactive whiteboard, global positioning system (GPS) unit, enterprise digital assistant (EDA), personal digital assistant (PDA), game console, or the like. The computing device100includes components of at least one processing element, some form of memory and input and/or output (I/O) devices. The components communicate with each other through inputs and outputs, such as connectors, lines, buses, cables, buffers, electromagnetic links, networks, modems, transducers, IR ports, antennas, or others known to those of ordinary skill in the art.

The computing device100has at least one display102for outputting data from the computing device such as images, text, and video. The display102may use LCD, plasma, LED, iOLED, CRT, or any other appropriate technology that is or is not touch sensitive as known to those of ordinary skill in the art. The display102may be co-located with at least one input surface104or remotely connected thereto. The input surface104may employ technology such as resistive, surface acoustic wave, capacitive, infrared grid, infrared acrylic projection, optical imaging, dispersive signal technology, acoustic pulse recognition, or any other appropriate technology as known to those of ordinary skill in the art to receive user input in the form of a touch- or proximity-sensitive surface. The input surface104may be bounded by a permanent or video-generated border that clearly identifies its boundaries. The input surface104may a non-touch sensitive surface which is monitored by a position detection system.

In addition to the input surface104, the computing device100may include one or more additional I/O devices (or peripherals) that are communicatively coupled via a local interface. The additional I/O devices may include input devices such as a keyboard, mouse, scanner, microphone, touchpads, bar code readers, laser readers, radio-frequency device readers, or any other appropriate technology known to those of ordinary skill in the art. Further, the I/O devices may include output devices such as a printer, bar code printers, or any other appropriate technology known to those of ordinary skill in the art. Furthermore, the I/O devices may include communications devices that communicate both inputs and outputs such as a modulator/demodulator (modem; for accessing another device, system, or network), a radio frequency (RF) or other transceiver, a telephonic interface, a bridge, a router, or any other appropriate technology known to those of ordinary skill in the art. The local interface may have additional elements to enable communications, such as controllers, buffers (caches), drivers, repeaters, and receivers, which are omitted for simplicity but known to those of skill in the art. Further, the local interface may include address, control, and/or data connections to enable appropriate communications among the other computer components.

The computing device100also includes a processor106, which is a hardware device for executing software, particularly software stored in the memory108. The processor can be any custom made or commercially available general purpose processor, a central processing unit (CPU), a semiconductor based microprocessor (in the form of a microchip or chipset), a macroprocessor, microcontroller, digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, state machine, or any combination thereof designed for executing software instructions known to those of ordinary skill in the art. Examples of suitable commercially available microprocessors are as follows: a PA-RISC series microprocessor from Hewlett-Packard Company, an 80x86 or Pentium series microprocessor from Intel Corporation, a PowerPC microprocessor from IBM, a Sparc microprocessor from Sun Microsystems, Inc., a 68xxx series microprocessor from Motorola Corporation, DSP microprocessors, or ARM microprocessors.

The memory108may include any one or a combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, or SDRAM)) and nonvolatile memory elements (e.g., ROM, EPROM, flash PROM, EEPROM, hard drive, magnetic or optical tape, memory registers, CD-ROM, WORM, DVD, redundant array of inexpensive disks (RAID), another direct access storage device (DASD)). Moreover, the memory108may incorporate electronic, magnetic, optical, and/or other types of storage media. The memory108can have a distributed architecture where various components are situated remote from one another but can also be accessed by the processor106. Further, the memory108may be remote from the device, such as at a server or cloud-based system, which is remotely accessible by the computing device100. The memory108is coupled to the processor106, so the processor106can read information from and write information to the memory108. In the alternative, the memory108may be integral to the processor106. In another example, the processor106and the memory108may both reside in a single ASIC or other integrated circuit.

The software in the memory108includes an operating system110and an application112. The software optionally further includes a handwriting recognition (HWR) system114which may each include one or more separate computer programs. Each of these has an ordered listing of executable instructions for implementing logical functions. The operating system110controls the execution of the application112(and the HWR system114). The operating system110may be any proprietary operating system or a commercially available operating system, such as WEBOS, WINDOWS®, MAC and IPHONE OS®, LINUX, and ANDROID. It is understood that other operating systems may also be utilized.

The application112includes one or more processing elements related to detection, management and treatment of hand-drawn shapes and handwritten text input by users (discussed in detail later). The software may also include one or more other applications related to handwriting recognition, different functions, or both. Some examples of other applications include a text editor, telephone dialer, contacts directory, instant messaging facility, computer-aided design (CAD) program, email program, word processing program, web browser, and camera. The application112, and the other applications, include program(s) provided with the computing device100upon manufacture and may further include programs uploaded or downloaded into the computing device100after manufacture.

The present system and method make use of the HWR system114in order to recognize handwritten input to the device100, including handwritten text and hand-drawn shapes, e.g., non-text. The HWR system114, with support and compliance capabilities, may be a source program, executable program (object code), script, application, or any other entity having a set of instructions to be performed. When a source program, the program needs to be translated via a compiler, assembler, interpreter, or the like, which may or may not be included within the memory, so as to operate properly in connection with the operating system. Furthermore, the handwriting recognition system with support and compliance capabilities can be written as (a) an object oriented programming language, which has classes of data and methods; (b) a procedure programming language, which has routines, subroutines, and/or functions, for example but not limited to C, C++, Pascal, Basic, Fortran, Cobol, Perl, Java, Objective C, Swift, and Ada; or (c) functional programing languages for example but no limited to Hope, Rex, Common Lisp, Scheme, Clojure, Racket, Erlang, OCaml, Haskell, Prolog, and F #. Alternatively, the HWR system114may be a method or system for communication with a handwriting recognition system remote from the device, such as server or cloud-based system, but is remotely accessible by the computing device100through communications links using the afore-mentioned communications I/O devices of the computing device100. Further, the application112and the HWR system114may operate together accessing information processed and stored in the memory108, for example, by each system, or be combined as a single application.

Strokes entered on or via the input surface104are processed by the processor106as digital ink. A user may enter a stroke with a finger or some instrument such as a pen or stylus suitable for use with the input surface. The user may also enter a stroke by making a gesture above the input surface104if technology that senses motions in the vicinity of the input surface104is being used, or with a peripheral device of the computing device100, such as a mouse or joystick. A stroke is characterized by at least the stroke initiation location, the stroke termination location, and the path connecting the stroke initiation and termination locations. Because different users may naturally write the same object, e.g., a letter, a shape, a symbol, with slight variations, the HWR system accommodates a variety of ways in which each object may be entered whilst being recognized as the correct or intended object.

FIG. 8is a schematic pictorial of an example of the HWR system114, in either its local (i.e., loaded on the device100) or remote (i.e., remotely accessible by the device100) forms. The HWR system114includes stages such as preprocessing116, recognition118and output120. The preprocessing stage116processes the digital ink to achieve greater accuracy and reducing processing time during the recognition stage118. This preprocessing may include normalizing of the path connecting the stroke initiation and termination locations by applying size normalization and/or methods such as B-spline approximation to smooth the input. The preprocessed strokes are then passed to the recognition stage118which processes the strokes to recognize the objects formed thereby. The recognized objects are then output120to the memory108and the display102as a digital ink or typeset ink versions of the handwritten elements/characters and hand-drawn shapes.

The recognition stage118may include different processing elements or experts.FIG. 9is a schematic pictorial of the example ofFIG. 8showing schematic detail of the recognition stage118. Three experts, a segmentation expert122, a recognition expert124, and a language expert126, are illustrated which collaborate through dynamic programming to generate the output120.

The segmentation expert122defines the different ways to segment the input strokes into individual element hypotheses, e.g., alphanumeric characters and mathematical operators, text characters, individual shapes, or sub expression, in order to form expressions, e.g., words, mathematical equations, or groups of shapes. For example, the segmentation expert122may form the element hypotheses by grouping consecutive strokes of the original input to obtain a segmentation graph where each node corresponds to at least one element hypothesis and where adjacency constraints between elements are handled by the node connections. Alternatively, the segmentation expert122may employ separate experts for different input types, such as text, drawings, tables, charts, equations, and music notation.

The recognition expert124provides classification of the features extracted by a classifier128and outputs a list of element candidates with probabilities or recognition scores for each node of the segmentation graph. Many types of classifiers exist that could be used to address this recognition task, e.g., Support Vector Machines, Hidden Markov Models, or Neural Networks such as Multilayer Perceptrons, Deep, Convolutional or Recurrent Neural Networks. The choice depends on the complexity, accuracy, and speed desired for the task.

The language expert126generates linguistic meaning for the different paths in the segmentation graph using language models (e.g., grammar or semantics). The expert126checks the candidates suggested by the other experts according to linguistic information130. The linguistic information130can include a lexicon(s), regular expressions, etc. The language expert126aims at finding the best recognition path. In one example, the language expert126does this by exploring a language model such as final state automaton (determinist FSA) representing the content of linguistic information130. In addition to the lexicon constraint, the language expert126may use statistical information modeling for how frequent a given sequence of elements appears in the specified language or is used by a specific user to evaluate the linguistic likelihood of the interpretation of a given path of the segmentation graph.

The application112provided by the present system and method allows users, such as students, academic and working professionals, to create handwritten diagrams and have those diagrams faithfully recognized using the HWR system114independent of the type of diagram created, e.g., flowcharts, organizational charts, concept maps, spider maps, block/architecture diagrams, mind-maps, block diagrams, Venn diagrams and pyramids. This list is not exhaustive and other types, or non-types, of diagrams are possible. For example, the different elements of the hand-drawn diagrams illustrated inFIGS. 1B, 2B3B4B,5B and6B are individually recognized together with any spatial and context relationships there between without regard to the diagram type. As discussed earlier, these diagram elements include shape and text elements. Shape or drawing elements are those that define graphic or geometric formations in linear or non-linear configurations, and include containers, connectors and free-form drawings. Text elements are those that contain text characters and include text blocks and labels for the text blocks and shape elements. Both text blocks and labels may contain text of one or more characters, words, sentences or paragraphs provided in one or more vertical lines and/or as numbered/bulleted lists. Text blocks may be contained by containers (internal text blocks) or may be provided outside of containers (external text blocks). External text blocks may be unrelated to containers or other elements of a diagram or may be directly related to certain other diagram elements.

Further, the application112provided by the present system and method allows users to hand-draw such shapes and text without any pre-determined or required order of drawing in order for proper recognition to be made. The handwriting recognition of the present system and method allows users to draw what they have in mind (freely without being slowed by the technology) as they would on paper, while benefiting from the power of digital tools. Example uses include:creation: high-level identification of, and differentiation between, shapes and text allows shapes to be sketched and text to be written by users without need to select or use pre-assigned tools nor switch between modes nor zoom to a specific area,editing: identification of shape features and points enables the shapes to be moved and manipulated for the creation of space for new ideas, the change of connections or shape type, and the handling of editing gestures, and the identification of text features and points enables the handling of editing gestures and the definition of text layout,searching: users can leverage the information contained in their diagrams to search through documents,typesetting: when creating diagrams users are provided with options for immediate recognition feedback,importing and exporting: identification of shape features and points and of text enables the use of data models suitable for the import of objects from, and the export of created documents to, processing and presentation tools.

These and other features of the present system and method and described in detail later.

Accurate content recognition of hand-drawn diagram elements is enabled by the application112through accurate and efficient differentiation of hand-drawn shapes and handwritten text. Further, fast and accurate recognition and provision of features, such as color-fill in shapes, is provided by the application112maintaining information on identified shape types (e.g., rectangle, circle, etc.) as recognized by the HWR system114rather than just information of the segmentation of the strokes making up the shapes used during the handwriting recognition process (for example, a rectangle is not stored by the application112as exploded into four lines).

Further, faithful rendering of hand-drawn diagrams is provided by the application112. This is because, the digital ink is retained (for example, in the memory108), the recognized shapes are not normalized, thereby maintaining the user defined size, aspect and form of the hand-drawn shapes, and shapes and text are able to be identified regardless of, or re-identified or classified based on, other elements of the diagram. That is, the application112identifies shapes that are hand-drawn alone (e.g., without associated connectors or text), as closed shapes or polygons (e.g., circles, ellipses, squares, rectangles and rhombi), and as open shapes (e.g., lines, which can be combined in the drawing of polygons). The application112further identifies shapes that are hand-drawn surrounding one or more existing non-text and/or text elements (e.g., creating an outline or container), creating containers, and creating connectors between other non-text and text elements. Containers that contain other shapes (including other containers) or text (e.g., by employing automatic grouping of elements) are also identified. The application112further identifies text that is handwritten alone (e.g., without an outline or container), within existing shapes (e.g., within a container), and near other elements. Text elements that contain a single line of text (one or several words) or multiple lines of text (with or without carriage return, numbered lists, bullets points, etc.) are also identified. The application112also allows text to be written directly in shapes without requiring an explicit action from the user or use a specific tool to trigger a dedicated input method. Text can also be input using non-handwriting techniques, such as with a keyboard connected to the computing device.

In editing, handwritten operations such as overwrite, erasure and layout control, can be performed on both digital and typeset ink. For example, overwriting includes changing a shape from one type or form to another (e.g., switching a rectangle to an ellipse by hand-drawing an ellipse over a rectangle), adding decoration to a connector, and creating a shape around text. Erasure can be performed using known handwriting gestures, such as scratch-out or strike-out gestures on the shapes and text. Layout control can be performed to move and resize shapes, align and distribute shapes and text to each other or one another. The detection of at least some of these operations, as well as others, is enhanced by the disambiguation process used by the present system and method. The disambiguation process is now described.

The present system and method automatically detect and differentiate the input of the different handwritten objects of shapes and text, so that they are processed by the HWR system114with suitable recognition techniques, e.g., the strokes of the detected shapes are processed using a shape language model and the strokes of the detected text are processed using a text language model. It is noted however that since many handwritten shapes and text characters can share common features (e.g., a circle and the letter “o”, an arrowhead and the letter “v”) users are provided with the ability to correct wrong differentiation decisions using the user interface (UI) of the application112.

The disambiguation process allows handwritten input containing mixed content of text and non-text (i.e., shapes) to be recognized and converted to beautified digital ink and typeset ink, either automatically (e.g., on-the-fly) or on demand. Digital ink is formed by rendering the handwritten input in digital image format. Beautified (digital) ink is formed by rendering the digital ink to appear more regular and normalized than the original handwriting while retaining similar styling or look-and-feel. Typeset ink is formed by converting the digital ink into typeset or fontified image format. Beautified typeset ink is formed by rendering the typeset ink with positional and styling changes from the input. The preprocessing stage116of the HWR system114is configured to perform the disambiguation process. The preprocessor116does this by classifying the elements of the digital ink into different classes or categories, being non-text (i.e., shape), text and a mixture of shape and text. The classified digital ink is then parsed to the recognizer118for suitable recognition processing depending on the classification.

For example, when processing digital ink classified as text, the recognizer118employs the segmentation expert122to segment individual strokes of the text to determine the segmentation graphs, the recognition expert124to assign probabilities to the graph nodes using the classifier128, and the language expert126to find the best path through the graphs using, for example, a text-based lexicon of the linguistic information130. On the other hand, when processing digital ink classified as non-text, the recognizer118employs the segmentation expert122to segment the strokes of the shape, the recognition expert124to determine segmentation graphs using the classifier128, and the language expert126to find the best path through the graphs using a shape-based lexicon of the linguistic information130. The mixed content classification is treated as ‘junk’ and will result in low probability of recognition when parsed to the recognizer118. Shapes that are parsed to the recognizer and not recognized because, for example, they are out-of-lexicon shapes are treated as doodles, being unrecognized content (described later).

FIG. 10Ashows an example hand-drawn diagram1000. Like the example diagrams described in the Background, the diagram1000has shape and text elements including diagram blocks or containers of different types (rectangles and a circle) and connectors of different types (straight arrows and a bent arrow) which connect or designate relationships between the diagram blocks, and text of different components (including single words, symbols and multiple words) within the containers.

FIG. 10Bis a flow diagram of an example disambiguation system and method, which is now described with respect to the example hand-drawn diagram1000. At step1, the disambiguator116receives the strokes of the hand-drawn input of the diagram1000, for example, strokes S1to S10.

At step2, the strokes are grouped using spatial and temporal considerations to build hypotheses of which strokes may belong to non-text or text elements of the diagram1000. The spatial considerations include distance between the strokes, geometry of the strokes, overlapping of the strokes, and relative positions of the strokes. For example, the strokes S1and S2are closer to one another, the strokes S1and S2surround the strokes S3to S8and the stroke S10overlays one end of the stroke S9. The temporal considerations include the time order of stroke input. For example, the strokes have the time order of S1, S2, S9, S10, S3, S4, S5, S7, S6and S8. The combined spatial and temporal considerations are used to provide a probability score to each grouping of strokes. The probability score may be a vector of three probability scores, one for each class, text, non-text and junk. A high probability score indicates that the group of strokes likely belongs to one object of the class associated with the score, a low probability score indicates that the group of strokes likely belong to separate objects or at least not to that the class associated with the score, with scores there between indicating relative likelihoods of classification. In this way, groups that achieve low or very low probability scores at this step are discarded such that hypotheses are not unnecessarily built for testing at the next step, thereby optimizing processing. This scoring defines a ‘coarse’ classification of the hypotheses to filter out those that should not be tested. Alternatively, hypotheses may be built for each classification for each systematic group of strokes, with classification of the strokes made on the class achieving the highest score.

The spatial and temporal information is also used in the building of the hypotheses themselves. For example, for a first group containing the strokes S1and S2and a second group containing the strokes S3to S8, as the spatial information is that the first group surrounds the second group and the temporal information is that the strokes of the first group were input in time order and the strokes of the second group were also input in time order, a first hypothesis is built that the first group belongs to one object, as it contains other elements, and a second hypothesis is built that the second group belongs to one object, as it is contained in another element, and a third hypotheses is built that the first and second groups belong to different objects. The third hypothesis is not merely a combination of the first and second hypotheses because it does not preclude the first or second group being of more than one object themselves.

For the examples of the strokes S1to S10, the following further groups have high probability scores and are therefore built as hypotheses:S3and S4; as they are time ordered and S4is closer to S3than S5,S5to S8; as they are time ordered,S2and S9; as they are time ordered and close spatially,S9and S10; as they are time ordered and S10overlays one end of S9.

On the other hand, a group with a medium probability score (e.g., neither likely nor unlikely) which is therefore likely used for building hypotheses contains the strokes S1and S9, as their spatial extents are near, but not particularly close, but they are close temporally, a group with a low probability score which is therefore likely not used for building hypotheses contains the strokes S1and S10, as they are far removed spatially with many other strokes there between, and a group with a very low probability score which is therefore not used for building hypotheses contains the strokes S5and S10, as they are far removed both spatially and temporally.

At step3, features for each group of strokes of the built hypotheses are extracted. These features are extracted in consideration of shape and text language models, which may be those implemented by the recognizer118or separately provided as part of the pre-processor116stored in the memory108, for example. The features include separation distances, changes in direction within strokes, overlap, direction of stroke pattern, relative geometrical extents, combined relative positions and time orders, curvature, linearity and looping. This list is not exhaustive, and about 100 different features are extracted by the present system and method, since the greater the number of features that are extracted, the greater the accuracy of the overall classification.

For example, for the first group described above, the extracted features of the stroke S1include two large changes in direction, three generally linear segments and partially closed formation, the extracted features of the stroke S2include single direction, generally linear, and the extracted relative features of the strokes S1and S2include horizontal extent of S2being within horizontal extent of S1and S2generally aligned with ends of two vertical segments of S1. For the second group described above, the extracted relative features of all the strokes include a generally unidirectional stroke pattern with some positionally delayed strokes (which may indicate the crossing of a “t” or dotting of an “i” character, for example), the extracted relative features of the strokes S5and S6include overlap, the extracted relative features of the strokes S5to S8include S6and S8positionally out of time order with S5and S7, and the extracted features of the stroke S7include many variations in direction (which may indicate cursive text, for example).

In Step4, the strokes are classified into text, non-text and junk by testing hypotheses based on all of the collected information, including extracted features of the groups of strokes within those hypotheses and the spatial and temporal information of the strokes within those groups, which is used to provide probability scores to each hypothesized group as a vector of three probability scores, one for each class, text, non-text and junk. For example, for the first and second hypotheses described above, the afore-mentioned extracted features and spatial and temporal information lead to the first group having a higher probability score for being non-text than for the other classifications of text and junk, and the second group having a higher probability score for being text than the other classifications of non-text and junk. Analysis of the probability scores may be done for each stroke, such that the probability score vectors for all of the hypotheses/groups containing a certain stroke are combined to provide an overall probability for that stroke as being text or non-text. This scoring defines a ‘fine’ classification of each of the strokes for recognition.

At step5, the classification results are parsed to the recognizer for handwriting recognition of the strokes by the appropriate recognition module.

Additionally, other disambiguation techniques employing simple heuristics can be used to supplement and augment the above-described technique. These include, for example, thresholds, writing speed, and time between strokes. The disambiguation system and method can be performed on already input handwriting or incrementally as handwriting is input. For incremental performance the probability scores required for confidence in classification can be set higher than for the non-incremental case, so that time is provided for input of all strokes of the objects before recognition is made. Alternatively, previously recognized objects may be re-recognized based on new input causing new groupings and hypotheses, however care is taken to not undesirably increase processing complexity and time (e.g., re-testing of all previously recognized elements) or cause upsetting re-recognition across the diagram being created or edited. For example, testing all recognized strokes which were entered before newly entered strokes, i.e., by time order, can quickly lead to over-processing which can affect recognition speed, thereby adversely affecting user experience of the application112. These non-beneficial effects may be minimized whilst providing good recognition of likely changed elements (due to new input) by restricting re-classification to certain strokes, such as, for example, testing only a restricted number of previously entered strokes, e.g., from one to about five strokes, on the basis of time order, or only testing previously recognized strokes which are spatially close to newly entered strokes for potential new groupings, or only testing recognized strokes of the same type, i.e., text or non-text, as the currently entered strokes upon classification of those strokes.

Accordingly, the non-text/text differentiation of the present system and method differs from previous techniques mentioned in the Background because any groups containing non-text and text strokes are given a low classification score and therefore not parsed to the recognizer, absolute classification is not performed prior to recognition rather hypotheses are built and tested, and segmented strokes themselves are not classified as belonging to non-text so as to be reformed into detectable shapes rather groups of strokes are classified and tested for shape recognition. As a result, the present system and method does not suffer from the potential for erroneous text and shape recognition and misclassification of the previous techniques.

Thus, the text/non-text disambiguator116of the present system is able to identify shapes and text, and within text, text lines, paragraphs defined as a coherent set of text lines, bulleted and numbered lists defined as a vertical set of text lines having initial bullet symbols or numbers, tables defined as a group of cells containing text, underlined text, and more. Further, the disambiguator116is able to identify the layout of a diagram, including shapes and text located inside shapes, text located inside a table cell and alignment of individual text blocks. Furthermore, the disambiguator116is able to identify corrected text, including scratch-outs defined as a scribbled line made over a piece of text or shape, strikethroughs defined as a single line made over a piece of text, and cross-outs defined as a pair of crossed lines, made over a piece of text. Further still, the disambiguator116is able to provide beautification information as the identified positions and sizes of the different elements can be adjusted to allow the recognized digital ink to be output in more regular and normalized form.

Operations and features of the application112in accordance with the present system and method are now described in relation to an example handwritten input of a diagram.FIG. 11Adepicts the digital ink rendering of a handwritten flow diagram1100input on the input surface104of the computing device100using the application112, andFIG. 11Bdepicts the flow diagram1100in typeset ink after recognition processing by the HWR system114. Like the example diagrams described in the Background, the diagram1100has shape and text elements including diagram blocks or containers12of different types (ovals, circles and diamonds) and connectors14of different types (straight arrows, bent arrows, open-headed arrows and closed-headed arrows) which connect or designate relationships between the diagram blocks12, and text16within some of the containers12, outside and associated with some of the containers12, and associated with some of the connectors14.

The manner in which the application112of the present system and method detects and identifies the handwritten elements of the diagram1100and displays the identified and recognized elements in digital and typeset ink is now described in relation toFIGS. 12 to 67.FIGS. 12 to 67show, in step-by-step fashion, the interface104of the computer device100having digital and typeset ink (after disambiguation and recognition by the HWR system114) rendered from hand-drawn input to the application112by a user with, for example, their finger or a stylus.

Each ofFIGS. 13 to 22 and 24 to 66are presented in an “A” version example in which typesetting is not automatically performed on the hand-drawn input such that the input is displayed as (non-beautified or beautified) digital ink and a “B” version example in which incremental typesetting is performed on the hand-drawn input such that the input is first displayed as digital ink and then as typeset ink. That is, the recognition processing of the present system and method may be performed in an incremental manner as handwritten input is received or may be performed as a batch process on already input content. In the latter case, the application112is configurable through selection by users, for example, through UI menus, buttons or performance of gestures, to render the typeset (or beautified) ink as the incremental recognition progresses, as depicted in the “B” version examples ofFIGS. 13 to 22 and 24 to 66. In the former case, display of the digital (or beautified) ink is maintained until selection by users to perform (beautification or) typesetting, as depicted in the “A” version examples ofFIGS. 13 to 22 and 24 to 66.

InFIG. 12, a hand-drawn shape1100is displayed as a digital ink shape1101a. The shape1101is hand-drawn in a single continuous stroke, detected as non-text and recognized as an oval or ellipse. Upon recognition as a closed shape, visual feedback may be provided to users (which is otherwise only ascertainable upon typesetting if incremental typesetting is not used) through specific rendering, such as color-fill within a digital ink shape.

InFIG. 13A, handwritten text1102input within the digital ink oval1101ais displayed as digital ink text1102a. InFIG. 13B, the recognition result ofFIG. 12is displayed as a typeset oval1101b, with the dimensions of the hand-drawn oval1101maintained, and the handwritten text1102input within the oval1101is displayed as the digital ink text1102awithin the typeset oval1101b. The text1102is detected as text and recognized as the word “Ideation”, and the oval1101is identified as a container which contains the recognized word1102due to the relative positions and characteristics of the inputs1101and1102. A container and text contained thereby are associated with one another so that some actions performed on one of these elements causes reactions on the other element. For example, when the container is selected and moved by the user the contained text is moved with the container, and when the text is selected and enlarged or added to by the user, the container is resized to accommodate the larger text size or block. Such operations are described in more detail later.

InFIG. 14A, a hand-drawn shape1103input beneath the oval container1101ais displayed as a digital ink shape1103a. InFIG. 14B, the recognition result ofFIG. 13Bis displayed as a typeset word1102bwithin the typeset oval container1102b, with the dimensions, and separation from the oval1101, of the handwritten word1102maintained, and the hand-drawn shape1103input beneath the typeset oval container1101bis displayed as the digital ink shape1103a. The shape1103is hand-drawn in two strokes, the grouped strokes are detected as non-text and recognized as a close-headed arrow due to the characteristics of the shape1103(i.e., a line dissecting one side and the opposite point of a triangle at one of its ends). An arrow is defined as being formed by a straight or bent ‘stem’ terminating in one or two open- (e.g., v-shaped) or closed- (e.g., triangular) ‘arrowheads’. At this point, the arrow1103may be identified as a connector of the oval1101due to their relative positions (e.g., a pre-set spatial separation threshold is used by the application112, where separation of an end of a linear shape to the non-linear shape below that threshold indicates a high likelihood of a connection relationship; the spatial threshold may be defined as a distance in pixels between mean points or barycenters of the strokes, for example, set to be about five pixels to about 100 pixels) and/or characteristics of the inputs1101and1103(e.g., an arrow having one end proximate or adjoining a container indicates a high likelihood of a connection relationship; with proximity defined in a distance range similar to the spatial threshold).

Alternatively, this identification may be deferred until another connected shape is input (i.e., a shape adjoining the other end of the arrow). Alternatively still or in addition, the arrow1103may be assigned with a certain probability of being a connector, such that the identification decision is quickly made upon the connected shape being input or upon the container1101, for example, being selected and moved by the user such that the identified connector1103is moved with the container1101. Such operations are described in more detail later. Upon identification as a connector, visual feedback may be provided to the user, such as a brief animation on the connected ends of the connector or rendering of the digital and typeset ink so that the connector ends ‘contact’ the boundaries of the shapes to which the connector connects.

InFIG. 15A, a hand-drawn shape1104input beneath the connector1104ais displayed as a digital ink shape1104a. InFIG. 15B, the recognition result ofFIG. 14Bis displayed as a typeset connector arrow1103b, with the proximate end of the hand-drawn arrow1103repositioned on the boundary of the typeset oval1103b, to provide visual feedback of the identified connector status, and the extent of distal end of the hand-drawn arrow1103from the typeset oval1101maintained, and the hand-drawn shape1104input beneath the connector1103is displayed as the digital ink shape1104a. The shape1104is hand-drawn in a single continuous stroke, detected as being non-text, recognized as being a diamond and identified as having one of its points connected to the connector1103due to the relative positions and/or characteristics of the inputs1103and1104. As discussed above, this identification may be used to decide on the identification of the connector1103. The typeset arrowhead of the connector1103may be rendered with a similar size to the hand-drawn arrowhead or a standardized arrowhead scaled (e.g., commensurate) with the dimensions (e.g., length, and width or ink weight, defined in pixels) of the connector may be used.

InFIG. 16A, handwritten text1105input within the diamond1104ais displayed as digital ink text1105a. InFIG. 16B, the recognition result ofFIG. 15Bis displayed as a typeset diamond1104b, with the dimensions of the hand-drawn diamond1104maintained and the upper point repositioned at the (relative) proximate end of the typeset connector1103b, and the handwritten text1105input within the diamond1104is displayed as the digital ink text1105awithin the typeset diamond1104b. The text1105is detected as being text and recognized as being the symbol “?”, and the diamond1105is identified as a container which contains the recognized symbol1105due to the relative positions and characteristics of the inputs1104and1105.

InFIG. 17A, handwritten text1106input to the left of the diamond1104ais displayed as digital ink text1106a. InFIG. 17B, the recognition result ofFIG. 16Bis displayed as a typeset symbol1105b, with the dimensions, and separation from the diamond1104, of the hand-drawn symbol1105maintained, and the handwritten text1106input to the left of the typeset diamond1104bis displayed as the digital ink text1106a. The text1106is detected as being text and recognized as being the words “No Go”. At this point, the text1106is identified as being associated with the container1104due to their relative positions (e.g., a pre-set, and re-settable, spatial separation threshold is used by the application112, where separation of geometric features of the detected text and non-text, such as mean centers or barycenters of the text and non-text blocks (defined by the x- and y-spatial extent of the text and non-text), adjacent or same positional (e.g., top) boundaries of the blocks, below that threshold indicates a high likelihood of associated objects, such as multiple associated shapes, multiple associated text blocks, and associated shapes and text blocks). This identified association means that if one of the elements1104or1106is selected and moved or edited, e.g., resized, by the user, the other element is influenced through relative movement or editing. Such operations are described in more detail later.

InFIG. 18A, a hand-drawn shape1107input beneath the words1106aand from another point of the diamond1104ais displayed as a digital ink shape1107a. InFIG. 18B, the recognition result ofFIG. 17Bis displayed as typeset words1106b, with the dimensions, and relative position to the diamond1104, of the hand-drawn words1106maintained, and the hand-drawn shape1107input beneath the typeset words1106band from the another point of the typeset diamond1104bis displayed as the digital ink shape1107a. The shape1107is hand-drawn in two strokes; the grouped strokes are detected as non-text and recognized as a close-headed arrow due to the characteristics of the shape1107. At this point, like the arrow1103, the arrow1107may be identified as a connector of the diamond1104. Further, at this point the arrow1107may be further identified as associated with the words1106due to their relative positions (determined by a pre-set separation threshold as described above) and/or characteristics of the inputs1106and1107(e.g., words positioned above an arrow), such that the text1106is defined as a label of the connector1107.

This identified association may be used to dissociate the container1104and words1106, for example, because the relative separation of the text1106from the connector1108is less than that from the container1104and/or because non-contained text (that is text not surrounded by a shape) is considered more likely to be associated with a proximate connector than other proximate shape types. Alternatively or in addition, the arrow1107and words1106may be assigned with a certain probability of being associated with one another, such that the identification decision is quickly made upon, for example, the arrow being selected and resized by the user such that the words1106are moved to retain their relative position to the arrow1107. Such operations are described in more detail later.

InFIG. 19A, a hand-drawn shape1108input to the left of the connector1107ais displayed as a digital ink shape1108a. InFIG. 19B, the recognition result ofFIG. 18Bis displayed as a typeset connector arrow1107b, with the proximate end of the hand-drawn arrow1107repositioned on the point of the typeset diamond1104b, and the extent of distal end of the hand-drawn arrow1107from the diamond1104and separation from the associated word1106maintained, and the hand-drawn shape1108input to the left of the typeset connector1107bis displayed as the digital ink shape1108a. The shape1108is hand-drawn in a single continuous stroke, detected as non-text, recognized as a circle and identified as having its boundary connected to the connector1107due to the relative positions and characteristics of the inputs1107and1108. As discussed above, this identification result may be used to decide on or affirm the connector status of the arrow1107. It is noted that the circle1108could easily have been detected as text rather than non-text, due to its similarity with the letter “o”. As discussed earlier, this is avoided by the grouping and probability scoring of the shapes1107and1108due to their relative positions (and perhaps temporal input) employed by the disambiguator116, or through correction mechanisms provided to users.

InFIG. 20A, handwritten text1109input to the left of the circle1108ais displayed as digital ink text1109a. InFIG. 20B, the recognition result ofFIG. 19Bis displayed as a typeset circle1108b, with the dimensions of the hand-drawn circle1108maintained and the proximate boundary point repositioned at the (relative) proximate end of the typeset connector1107b, and the handwritten text1109input to the left of the typeset circle1108bis displayed as the digital ink text1109a. The text1109is detected as text and recognized as the word “Stop”. At this point, the text1106is likely identified as being associated with the shape1108due to their relative positions, like the container1104and the words1106at the point ofFIG. 17, and as such the text1109is defined as a label of the container1108.

InFIG. 21A, a hand-drawn shape1110input to the right of the shapes1101aand1104afrom another point of the diamond1104ato another point on the boundary of the oval1101ais displayed as a digital ink shape1110a. InFIG. 21B, the recognition result ofFIG. 20Bis displayed as a typeset word1109b, with the dimensions, and relative position to the typeset circle1108b, of the hand-drawn word1109maintained, and the hand-drawn shape1110input to the right of the typeset shapes1101band1104bfrom another point of the typeset diamond1104bto another point on the boundary of the typeset oval1101bis displayed as the digital ink shape1110a. The shape1110is hand-drawn in a single continuous stroke, detected as non-text and recognized as a bent line due to the characteristics of the shape1110. At this point, because the ends of the line respectively adjoin the containers1101and1104, the bent line1110is identified as a connector connecting these containers. This identification result is made without the presence of a connector element, such as an arrowhead. This is because the above-described spatial threshold for the high likelihood of connector detection is satisfied regardless of the presence of such connector elements.

InFIG. 22A, a hand-drawn shape1111input over the end of the connector1110aproximate the oval1101ais displayed as a digital ink shape1111a. InFIG. 22B, the recognition result ofFIG. 21Bis displayed as a typeset bent connector1110b, with one end of the hand-drawn connector1110repositioned on the point of the typeset diamond1104band the other end of the hand-drawn connector1110repositioned on the boundary of the typeset oval1101b, and the dimensions of the bends within the hand-drawn line1110maintained, and the hand-drawn shape1111input over the end of the typeset connector1110bproximate the typeset oval1101bis displayed as the digital ink shape1111a. The shape1111is hand-drawn in a single continuous stroke, detected as non-text and recognized as a closed arrowhead due to the characteristics of the shape1111(i.e., a triangle having one side and the opposite point intersected by the line1110at one of its ends). This determination of an arrowhead added to the line1110can be used to affirm the connector status of the line1110, for example, by an increase in an overall probability score, as follows.

Each of the arrows1103,1107and1110are hand-drawn using two strokes. However, the input timing of the individual strokes of each arrow differs such that the returned recognition of the HWR system114with respect to typesetting of the input differs. That is, the strokes of the arrows1103and1107are drawn within a relatively short space of time, say, within one second, such that they are classed together as non-text by the disambiguator116for parsing to the recognizer118(e.g., a pre-set, and re-settable, temporal separation threshold is used by the application112, where the time separation between drawing strokes below that threshold indicates a high likelihood of the strokes being parts of a single shape), and as such the typeset arrows1103band1107bare output by the HWR system114. On the other hand, the strokes1110and1111of the arrow1110are drawn within a relatively long space of time, say, more than one second (e.g., greater than the temporal separation threshold), such that the stroke1110is parsed to the recognizer118alone and output as the typeset line1110b. The proper identification of the complete connector arrow1110is provided however through the overall probability score calculated from spatial and temporal scores and metrics or characteristics of the input strokes. That is, while the temporal score for the strokes1110and1111being part of the same object is low, the combination of the high spatial score for the strokes1110and1111being part of the same object, based on the characteristics (i.e., the line1110dissecting the shape1111) result in the overall probability of the strokes1110and1111belonging to one shape being high.

This is illustrated inFIG. 23for the incremental typesetting example, where the recognition result ofFIG. 22Bis displayed as a typeset connector arrowhead1111b, with the point of the hand-drawn arrowhead1111repositioned on the end of the typeset line1110band on the boundary of the typeset oval1101b, thereby reforming the typeset connector1110bas a connector arrow as a result of the above-described processing.

InFIG. 24A, handwritten text1112input below the connector arrow1110ais displayed as digital ink text1112a. InFIG. 24B, the handwritten text1112input below the typeset connector arrow1110bof the recognition result ofFIG. 23is displayed as the digital ink text1112a. The text1112is detected as text and recognized as the word “REDIRECT”. The text1112is identified as being associated with the connector1110due to their relative positions.

InFIG. 25A, handwritten text1113input to the right of the connector1103ais displayed as digital ink text1113a. InFIG. 25B, the recognition result ofFIG. 24Bis displayed as a typeset word1112b, with the dimensions, and relative position to the typeset connector1110b, of the hand-drawn word1112maintained. Accordingly, it can be seen from the detected inputs and associations ofFIGS. 17 and 18, andFIGS. 24 and 25, that the identification of connectors and associated text (e.g., connector labels) is made by the present system and method regardless of the order in which these elements are input. Further inFIG. 25B, the handwritten text1113input to the right of the typeset connector1103bis displayed as the digital ink text1113a. The text1113is detected as being text and recognized as being the term “S-Gate”. The text1113is identified as being associated with the connector1103due to their relative positions (e.g., the text1113is not contained in a shape and is closer to the connector1103than any of the other elements of the diagram1100).

InFIG. 26A, a hand-drawn shape1114input beneath the diamond1104afrom another point of the diamond1104ais displayed as a digital ink shape1114a. InFIG. 26B, the recognition result ofFIG. 25Bis displayed as a typeset term1113b, with the dimensions, and relative position to the typeset connector1103b, of the hand-drawn term1113maintained, and the hand-drawn shape1114input beneath the typeset diamond1104bfrom another point of the typeset diamond1104bis displayed as the digital ink shape1114a. The shape1110is hand-drawn in two strokes; the grouped strokes are detected as non-text and recognized as an open-headed arrow due to the characteristics of the shape1114. As before, the arrow1114may be determined at this point to be a connector due to its relationship with the container1104and its arrow characteristic.

InFIG. 27A, handwritten text1115input to the left of the connector arrow1114ais displayed as digital ink text1115a. InFIG. 27B, the recognition result ofFIG. 26Bis displayed as a typeset connector arrow1114b, with the proximate end of the hand-drawn arrow1114repositioned on the point of the typeset diamond1104b, and the extent of distal end of the hand-drawn arrow1114from the typeset diamond1104bmaintained, and the handwritten text1115input to the left of the typeset connector arrow1114bis displayed as the digital ink text1115a. The text1115is detected as text and recognized as the word “Go”. The text1115is identified as being associated with the connector1114due to their relative positions.

InFIG. 28A, handwritten text1116input below the connector1114ais displayed as digital ink text1116a. InFIG. 28B, the recognition result ofFIG. 27Bis displayed as a typeset word1115b, with the dimensions, and relative position to the typeset connector1114b, of the hand-drawn word1115maintained, and the handwritten text1116input below the typeset connector1114bis displayed as the digital ink text1116a. The text1116is detected as text and recognized as the word “Initialization”. At this point, the text1116may be identified as being associated with the connector1114due to their relative positions (e.g., determined by the separation threshold) and/or characteristics of the inputs1114and1116(e.g., words positioned below the head of an arrow). This detected association is performed without the presence of a container for the text1116.

InFIG. 29A, a hand-drawn shape1117input to surround the word1116ais displayed as a digital ink shape1117a. InFIG. 29B, the recognition result ofFIG. 28Bis displayed as a typeset word1116b, with the dimensions, and relative position to the typeset connector1114b, of the hand-drawn word1116maintained. Accordingly, it can be seen from the detected inputs and associations ofFIGS. 12 and 13, andFIGS. 28 and 29, that the identification of containers and contained text is made by the present system and method regardless of the order in which these elements are input. Further, the hand-drawn shape1117input to surround the typeset word1116bis displayed as the digital ink shape1117a. The shape1117is hand-drawn in a single, continuous stroke, detected as non-text, identified as a container which contains the word1116due to the relative positions and characteristics of the inputs1116and1117, and recognized as an oval. The determination of the container1117can be used to affirm the connector status of the arrow1114, for example, by an increase in the overall probability score due to the arrow1114being positioned between the containers1104and1117.

InFIG. 30A, a hand-drawn shape1118input beneath the container1117ais displayed as a digital ink shape1118a. InFIG. 30B, the recognition result ofFIG. 29Bis displayed as a typeset oval1117b, with the dimensions of the hand-drawn oval1117maintained and the separation from the head of the typeset arrow1116maintained (this is done here to show that the rendering of all of different types of connectors, e.g., close-headed arrows versus open-headed arrows, need not be adjusted to actually ‘connect’ with the connected shapes in order to be considered a connector) and the hand-drawn shape1118input beneath the typeset container1117is displayed as the digital ink shape1118a. The shape1118is hand-drawn in a single continuous stroke, detected as non-text and recognized as an open-headed arrow due to the characteristics of the shape1118. As before, the arrow1118may be determined at this point to be a connector due to its arrow characteristic and its relationship with the container1117.

InFIG. 31A, a hand-drawn shape1119input beneath the connector1104aoverlaying part of the arrow1118ais displayed as a digital ink shape1119a. InFIG. 31B, the recognition result ofFIG. 30Bis displayed as a typeset connector arrow1118b, with the proximate end of the hand-drawn arrow1118repositioned on the typeset oval1117and the extent of distal end of the hand-drawn arrow1118from the typeset oval1117maintained, and the hand-drawn shape1119input beneath the typeset connector1104overlaying part of the typeset arrow1118is displayed as the digital ink shape1119a. The shape1119is hand-drawn in a single continuous stroke, detected as non-text, recognized as a diamond and identified as having its upper and lower points intersected by the connector1118due to the relative positions and/or characteristics of the inputs1118and1119. At this point, the determination of the diamond1119as a (probable) container and a connection association between the diamond1119and the (probable) connector arrow1118may be assigned low probability scores since the diamond1119is not positioned proximate the ‘free’ end of the connector1118(e.g., at the arrowhead).

InFIG. 32A, a hand-drawn input1120within the diamond1119aand overlaying the portion of the connector1118athere within is displayed as digital ink1120a. InFIG. 32B, the recognition result ofFIG. 31Bis displayed as a typeset diamond1119b, with the dimensions of the hand-drawn diamond1119maintained and the upper and lower points repositioned on the typeset connector1118, and the hand-drawn input1120within the typeset diamond1119and overlaying the portion of the typeset connector1118there within is displayed as the digital ink1120awithin the typeset diamond1119b. The input1120is detected as a handwritten editing gesture and identified as a scratch-out on the stem portion of the arrow1118within the diamond1119. This scratch-out editing operation is interpreted by the application112as an erasure of the portion of the arrow1118within the diamond1119b. As a result, the arrow1118is split into two shapes, one a line between the container1117and the top point of the diamond1119and the other an arrow projecting from the bottom point of the diamond1119. At this point, the line is identified as a connector between the container1117and the diamond1119and the arrow is identified as a likely connector due to its arrow characteristic and its relationship with the shape1119, with consequential detection of the diamond1119as likely being a container.

InFIG. 33A, the erasure identification causes the digital ink arrow1118ato be split into a line connector1121displayed as digital ink line1121abetween the containers1117aand1119aand an arrow1122displayed as digital ink arrow1122aprojecting from the bottom of the container1119a. Further, a hand-drawn shape1123input over the end of the connector1121aproximate the diamond1119ais displayed as a digital ink shape1123a. InFIG. 33B, the erasure identification causes the typeset arrow1118bto be split into the line connector1121displayed as a typeset line connector1121bbetween the typeset containers1117band1119band the arrow1122displayed as a typeset arrow1122bprojecting from the bottom of the typeset container1119b. Further, the hand-drawn shape1123input over the end of the typeset connector1121proximate the typeset diamond1119bis displayed as the digital ink shape1123a. The shape1123is hand-drawn in a single continuous stroke, detected as non-text and recognized as a closed arrowhead due to the characteristics of the shape1123. This determination of an arrowhead added to the line1121can be used to affirm the connector status of the line1121in the manner described earlier. Further, at this point the newly split arrow1122may be determined to be a connector due to its relationship with the container1119and its arrow characteristic.

InFIG. 34A, handwritten text1124input within the diamond1119ais displayed as digital ink text1124a. InFIG. 34B, the recognition result ofFIG. 33Bis displayed as a typeset connector arrowhead1123b, with the point of the hand-drawn arrowhead1123repositioned on the end of the typeset line1121band on the top point of the typeset diamond1119b, thereby reforming the typeset connector1121bas a connector arrow as a result of the above-described processing. Further, the handwritten text1124input within the typeset diamond1119bis displayed as the digital ink text1124awithin the typeset diamond1119b. The text1124is detected as text and recognized as the symbol “?”, and the identification of the diamond1119as being a container is affirmed as it contains the recognized symbol1124.

InFIG. 35A, a hand-drawn shape1125input from another point of the diamond1119ais displayed as a digital ink shape1125a. InFIG. 35B, the recognition result ofFIG. 34Bis displayed as a typeset symbol1124b, with the dimensions, and separation from the typeset diamond1119b, of the hand-drawn symbol1124maintained, and the hand-drawn shape1125input from the point of the typeset diamond1119bis displayed as the digital ink shape1125a. The shape1125is hand-drawn with multiple strokes, detected as non-text and recognized as a closed-headed arrow due to the characteristics of the shape1125. As before, the arrow1125may be determined at this point to be a connector due to its arrow characteristic and its relationship with the container1119.

InFIG. 36A, a hand-drawn shape1126input to the left of the connector1125ais displayed as a digital ink shape1126a. InFIG. 36B, the recognition result ofFIG. 35Bis displayed as a typeset connector arrow1125b, with the proximate end of the hand-drawn arrow1125repositioned on the proximate point of the typeset diamond1119band the extent of distal end of the hand-drawn arrow1125from the typeset container1119bmaintained, and the hand-drawn shape1126input to the left of the typeset connector1125bis displayed as the digital ink shape1126a. The shape1126is hand-drawn in a single continuous stroke, detected as non-text, recognized as a square and identified as having its boundary connected to the connector1125due to the relative positions and characteristics of the inputs1125and1126. As discussed earlier, this identification result may be used to affirm the connector status of the arrow1125.

The previously described recognized closed-headed arrows include hand-drawn non-filled or un-filled triangles as arrowheads which are rendered by the application112into non-filled closed arrowheads in the digital ink and into filled closed arrowheads in the typeset ink (see, for example,FIGS. 14 and 15). In contrast, the closed-headed arrow1125includes a hand-drawn, filled (using multiple strokes or a single continuous stroke with many changes in direction) triangle as an arrowhead which is rendered by the application112into a filled closed arrow-head in the digital ink (seeFIG. 35A) and into a filled closed arrowhead in the typeset ink (seeFIG. 36B). Accordingly, the typeset result is the same. Alternatively, non-filled arrowheads can be retained in the typeset ink, as selectable by users through the UI of the application112.

InFIG. 37A, a hand-drawn shape1127is displayed as a digital ink shape1127aoverlaying the square1126a. InFIG. 37B, the recognition result ofFIG. 36Bis displayed as a typeset square1126b, with the dimensions of the hand-drawn square1126maintained and the proximate boundary point repositioned at the (relative) proximate end of the typeset connector1125b, and the hand-drawn shape1127is displayed as the digital ink shape1127aoverlaying the typeset square1126b. The shape1127is hand-drawn in a single continuous stroke, detected as non-text and recognized as a circle. The detection of a shape substantially overlaying a previously identified shape is interpreted by the application112as an overwrite or replacement gesture. For example, a pre-set, and re-settable, geometric threshold is used by the application112, where the average or mean distance between linear points, boundaries or peripheries of overlaid geometric shapes less than that threshold indicates a high likelihood of an overwrite gesture. Accordingly, the overwritten shape is deleted or omitted from display (together with or without text contained thereby, for example) and replaced with the new shape.

InFIG. 38A, the detected overwrite gesture causes replacement of the square1126awith a circle1127displayed as a digital ink circle1127aand handwritten text1128input above the connector arrow1125ais displayed as digital ink text1128a. InFIG. 38B, the detected overwrite gesture causes replacement of the typeset square1126bwith the circle1127displayed as a typeset circle1127b, with the dimensions of the hand-drawn circle1127maintained and the proximate boundary point repositioned at the (relative) proximate end of the typeset connector1125b, and the handwritten text1128input above the typeset connector arrow1125bis displayed as the digital ink text1128a. The text1128is detected as text and recognized as the words “No Go”. The text1128is identified as associated with the connector1125due to their relative positions.

InFIG. 39A, a hand-drawn input1129is displayed as digital ink1129aoverlaying the words1128a. InFIG. 39B, the recognition result ofFIG. 38Bis displayed as typeset words1128b, with the dimensions, and relative position to the typeset connector1125b, of the hand-drawn words1128maintained, and the hand-drawn input1129is displayed as the digital ink1129aoverlaying the typeset words1128b. The input1129is detected as a handwritten editing gesture and identified as a scratch-out on the words1128. This scratch-out editing operation is interpreted by the application112as an erasure of the words1128.

InFIG. 40A, the detected erasure causes removal of the digital ink words1128afrom display and inFIG. 40B, the detected erasure causes removal of the typeset words1128bfrom display.

InFIG. 41A, a gesture input1130is detected and identified as selection of the label1106aof the connector1107athereby causing display of a selection box1131surrounding the label1106areplaced by display of a selected typeset label1106c, having selection mode properties described below, thereby providing the user with visual feedback of the identified gesture. The gesture1130is, for example, a long tap on the interface surface104of the computing device100as depicted inFIG. 41. Such a gesture and its relative timing of “long” are well understand by one of ordinary skill in the art.

InFIG. 41B, the detection of the selection gesture causes display of the selection box1131surrounding the typeset label1106balso replaced by display of the selected typeset label1106c. The selection box1131is displayed with handles1131afor providing interaction operations on the content. Depending of the selection gesture employed, different selection modes can be initiated. For example, the long press selection gesture illustrated inFIG. 40is detected by the application112as initiating a multi-selection mode, i.e., the selection of multiple elements, such that the application112expects subsequent selection of other diagram elements.

Accordingly inFIG. 41A, a subsequent gesture input1132is detected and identified as selection of the label1109aof the container1108aand inFIG. 41Bdetected and identified as selection of the typeset label1109bof the typeset container1108b. The gesture1132is, for example, a short tap on the interface surface104of the computing device100as depicted inFIG. 41. Such a gesture and its relative timing of “short” are well understand by one of ordinary skill in the art. While a selection mode is active, the selected elements of the diagram1110are displayed as typeset versions which have different properties to the un-selected typeset versions (e.g., a different color and/or greater ink weight). Further, the non-selected elements of the diagram1100may be displayed differently to the normal rendering (e.g., with reduced opacity or ink weight, say at about 20% to about 50%, as shown inFIG. 41).

As discussed above, during selection of shapes, including containers and connectors, and text temporary selection mode display is made to provide visual feedback to users of the detected selection. The selection mode display can be rendered in a number of ways. For example, both digital ink and typeset ink may be displayed as selected typeset ink having the selection mode properties described above. Alternatively, typeset ink may be displayed as selected typeset ink and for the digital ink, both the digital ink and the corresponding typeset ink may be displayed together, with the typeset ink displayed as selected typeset ink and the digital ink displayed with reduced emphasis, like the non-selection properties described above. Several examples of this selection mode rendering are shown inFIGS. 68 to 72.

InFIG. 68A, a hand-drawn shape is displayed as a digital ink shape680with no other shape or text content. The shape680is detected as non-text and recognized as a triangle. Further, a gesture input682made within the triangle680is detected and identified as selection of the triangle680. The gesture682is, for example, a long tap on the interface surface104of the computing device100as depicted inFIG. 68A.

InFIG. 68B, the detection of the selection gesture causes display of a digital ink triangle680awith reduced emphasis (e.g., greyed-out) and overlaid with a selected typeset ink triangle684rendered with heightened emphasis (e.g., with a different color and color-filling within the triangle) and bounded by a bounding (selection) box686(the color-filling may be within the entire bounding box rather than just the bounded shape). In this way, users are provided with feedback of the selection detection and a sense of the typesetted extent of the hand-drawn shape, to guide movement and re-sizing operations for example, while maintaining display of the original input content.

InFIG. 69A, a hand-drawn shape and text are displayed as a digital ink shape690and digital ink text692, respectively. The shape690is detected as non-text and recognized as a circle, the text692is detected as text and recognized as the word “Text”, and the shape690is identified as a container containing the word692. Further, a gesture input694made on the word692is detected and identified as selection of the word692. The gesture694is, for example, a long tap on the interface surface104of the computing device100as depicted inFIG. 69A.

InFIG. 69B, the detection of the selection gesture causes display of a digital ink word692awith reduced emphasis (e.g., greyed-out) and overlaid with a selected typeset ink word696rendered with heightened emphasis (e.g., with a different color) and bounded by a bounding (selection) box698(with color-filling), while display of the digital ink circle690is maintained. In this way, users are provided with feedback of the selection detection and a sense of the typesetted extent of the handwritten text, to guide movement and re-sizing operations for example, while maintaining display of the original input content.

InFIG. 70A, a hand-drawn shape and text are displayed as a digital ink shape700and digital ink text702, respectively. The shape700is detected as non-text and recognized as an ellipse, the text702is detected as text and recognized as the word “Initialization”, and the shape700is identified as a container containing the word702. Further, a gesture input704made on the boundary or periphery of the ellipse700is detected and identified as selection of the ellipse700. The gesture704is, for example, a long tap on the interface surface104of the computing device100as depicted inFIG. 70A.

InFIG. 70B, the detection of the selection gesture causes display of a digital ink ellipse700awith reduced emphasis (e.g., greyed-out) and overlaid with a selected typeset ink ellipse706rendered with heightened emphasis (e.g., with a different color and color-filling within the ellipse) and bounded by a bounding (selection) box708, while display of the digital ink word702is maintained. In this way, users are provided with feedback of the selection detection and a sense of the typesetted extent of the handwritten shape, to guide movement and re-sizing operations for example, while maintaining display of the original input content.

InFIG. 71A, the digital ink circle690and digital ink word692ofFIG. 69Aare displayed. A gesture input710made within the container690but not on the word692is detected and identified as selection of the entire container, that is, both the circle690and the text692. The gesture710is, for example, a long tap on the interface surface104of the computing device100as depicted inFIG. 71A.

InFIG. 71B, the detection of the selection gesture causes display of a digital ink circle690aand the digital ink word692awith the reduced emphasis and overlaid with a selected typeset ink circle712and the selected typeset ink word696rendered with the heightened emphasis and bounded by a bounding (selection) box714. In this way, users are provided with feedback of the selection detection and a sense of the typesetted extent of the entire handwritten container, to guide movement and re-sizing operations for example, while maintaining display of the original input content.

InFIG. 72A, a hand-drawn shape is displayed as a digital ink shape720, handwritten text is displayed as digital ink text721, and hand-drawn input is displayed as a digital ink input722. The shape720is detected as non-text and recognized as an ellipse, the text721is detected as text and recognized as the word “Idea”, the input722is detected as either non-text or as junk, and the shape720is identified as a container containing the word721and the input722. A gesture input723made within the container720but not on the word721or input722is detected and identified as selection of the entire container, that is, the ellipse700, the text721and the input722. The gesture723is, for example, a long tap on the interface surface104of the computing device100as depicted inFIG. 72A.

InFIG. 72B, the detection of the selection gesture causes display of the digital ink ellipse720aand the digital ink word721a with the reduced emphasis and overlaid with a selected typeset ink ellipse724and a selected typeset ink word725rendered with the heightened emphasis and bounded by a bounding (selection) box726. However, the input722is either not recognizable by the recognizer118if classified as non-text or not parsed to the recognizer if classified as junk. Accordingly, the unrecognized input721is displayed as selected digital ink727, which is the digital ink input722rendered with the heightened emphasis, as a doodle. In this way, users are provided with feedback of the selection detection and a sense of the typesetted extent of the entire handwritten container, to guide movement and re-sizing operations for example, while maintaining display of the original input content.

InFIGS. 69 to 72, the type of selection of the diagram elements detected depends on the detected position of the selection gesture in relation to the diagram elements. That is, for the containers containing text of these examples, depending on the gesture position either the container only, the text only, or the container and its contained elements (either or both of non-text and text) is selected. Other selection behavior is possible however. For example, a selection gesture at any position within or on a container and its contained elements may cause selection of the entire container. Further, differentiation between containers, which may contain other shapes, and cells, which are containers that only contain text, may be made. Such that a selection gesture at any position within or on a cell and its contained text may cause selection of the entire cell, whereas like selections made with a container cause the selections shown inFIGS. 69 to 72.

The above-described overlay of selected typeset ink over the digital ink display is also applicable to other selected diagram elements, in particular, connectors. Alternatively or in addition to these selection mode examples, similar overlaid display of digital and typeset ink may be provided, albeit without the selection mode rendering, to allow users to preview the typesetted version of the digital ink without typesetting the diagram elements per se. The preview mode is initiated, for example, upon detection of a preview gesture, such as short tap or ‘hover’ of a user's finger or stylus over a diagram element.

Returning to the example input, inFIG. 42A, the detection of the subsequent selection gesture causes extended display of the selection box1131about the selected typeset label1106cand the label1109areplaced by display of a selected typeset label1109c, having the selection mode properties, thereby providing the user with visual feedback of the identified gesture. InFIG. 42B, the detection of the subsequent selection gesture causes the extended display of the selection box1131surrounding the selected typeset label1106cand the typeset label1109balso replaced by display of the selected typeset label1109c. Further gesture inputs1132are detected and identified as selections of the label1113aof the connector1103a, the label1112aof the connector arrow1110aand the connector arrow1110aitself inFIG. 42Aand the typeset label1113bof the typeset connector1103b, the typeset label1112bof the typeset connector arrow1110band the typeset connector arrow1110bitself inFIG. 42B.

InFIG. 43A, the detection of the further selection gestures causes further extension of the selection box1131about the multi-selected elements1103a,1106a,1109a,1110a,1112aand1113aand a selected typeset label1112c, a selected typeset label1113cand a selected typeset connector1110crespectively for the labels1112aand1113aand the connector1110a, having the selection mode properties, thereby providing the user with visual feedback of the recognition. InFIG. 43B, the detection of the further selection gestures causes the further extension of the selection box1131about the multi-selected typeset elements1103b,1106b,1109b,1110b,1112band1113band the selected typeset labels1112cand1113cand connector1110crespectively for the typeset labels1112band1113band the typeset connector1110b.

InFIGS. 44A and 44B, a gesture input1133is detected as selection of the selection box1131. The gesture1133is, for example, a long tap on the interface surface104of the computing device100within the selection box1131but not on any of the diagram elements therein, as depicted inFIG. 44. In response to this recognized gesture, the application112displays a selection menu1134displaying certain editing options that can be performed on the selected elements within the selection box1131. For example, the options of Duplicate, Copy and Paste is provided by the menu. This provision of the ‘pop-up’ or on-demand menu is merely an example, and other mechanisms for providing editing operations can be used, such as different gestures for different operations or only a default operation being selected through the use of a gesture.

InFIGS. 45A and 45B, a gesture input1135is detected as selection of the Duplicate operation from the menu1134causing the selected elements of the labels1106a,1109a,1112aand1113aand the connector arrow1110ainFIG. 45Aand typeset labels1106b,1109b,1112band1113band the typeset connector arrow1110binFIG. 45Bto be duplicated. The duplicate elements are retained in the selection box1131in the selected mode rendering with the same positional relationships as the original elements that have been duplicated, however the display of the elements that have been duplicated returns to the normal rendering or reverts to the non-selected rendering, with reduced emphasis, like inFIG. 47. The gesture1130is, for example, a press on the interface surface104of the computing device100as depicted inFIG. 45. Further, the duplication operation can be configured to omit any selected text so that the structure of a diagram can be copied without having to erase/replace the labels, for example.

InFIGS. 46A and 46B, a gesture input1136is detected as a move operation on the selection box1131in the direction of arrow A. The gesture1136is, for example, a long tap-and-slide (such as a drag or push) on the interface surface104of the computing device100within the selection box1131but not on any of the diagram elements therein, as depicted inFIG. 46.

InFIGS. 47A and 47B, completion of the move operation is detected and display of the selected typeset label1106cpositioned adjacent the container1127aor1127b, the selected typeset label1109cpositioned adjacent the connector1125aor1125b, the selected typeset label1112cpositioned adjacent the connector1121aor1121b, the selected typeset label1113cpositioned adjacent the connector arrow1110aor1110band the selected typeset connector arrow1110cpositioned to connect the containers1117aand1119aor1117band1119bis made, respectively. Accordingly, already drawn and recognized diagram elements can be re-used in or moved to other parts of the diagram, or on other diagrams as the diagrams are being created. In this process, the duplicate arrow1110is identified as a connector connecting the containers1117and1119due to the connector status of the original connector arrow1110, the container status of the containers1117and1119, and otherwise due to the relative positions and characteristics of these elements, in the manner described earlier.

InFIGS. 48A and 48B, a gesture input1137is detected as a de-selection gesture and in response display of the selection box1131is omitted and the selection mode typeset display of the de-selected diagram elements is omitted with display reverting to the normal digital ink or typeset display versions, and the display of the other non-selected diagram elements is returned to the normal rendering, if applicable for the multi-selection mode. The gesture1137is, for example, a short tap on the interface surface104of the computing device100outside of the selection box1131, as shown inFIG. 48.

InFIG. 49A, a gesture input1138is detected as selection of the duplicate connector arrow1110a. InFIG. 49B, the gesture input1138is detected as selection of the duplicate typeset connector arrow1110b. The gesture1133is, for example, a short tap on the interface surface104of the computing device100as depicted inFIG. 49. Accordingly, the short tap selection gesture is identified by the application112as initiating a uni-selection mode, i.e., the selection of a single diagram element such as shown in the example ofFIG. 68, such that the application112expects subsequent operation performed on the selected element.

InFIG. 50A, in accordance with the selection detection, the connector arrow1110ais replaced by display of a selected typeset arrow1110chaving the selection mode properties, thereby providing the user with visual feedback of the detection. InFIG. 50B, in accordance with the selection detection, the typeset connector arrow1110bis replaced by display of the selected typeset arrow1110c. Unlike for the multi-select mode described above, a selection box is not also used in the connector selection mode in order to provide clear display of the connector during the users' operations thereon.

InFIGS. 51A and 51B, a gesture input1139is detected as a move operation on the bent connector arrow1110cat the arrowhead and/or portion of the connector arrow stem having the arrowhead in the direction of arrow B. The gesture1139is, for example, a long tap-and-slide on the interface surface104of the computing device100as depicted inFIG. 51. As the connector arrow1110cis a connector connected to the containers1117and1119, the move gesture is identified as only pertaining to the arrowhead stem portion connected to the container1117, such that the end of the connector arrow1110cconnected or anchored to the container1119is not affected during the move operation. During the move operation therefore the connector arrow1110is decoupled from the container1117and resizing and re-display of certain portions of the bent connector arrow1110care made, chiefly the horizontal stem portion connecting the vertical stem portions connected to the respective containers is resized to accommodate the movement of the arrowhead stem. During such move operations, a specific visual feedback may be displayed, such as a ‘lens’, to guide interaction with the objects, particularly, if the user's finger is used.

InFIGS. 52A and 52B, completion of the move operation is detected and display of the connection or anchor point of the arrowhead of the bent connector arrow1110cat the boundary of the container1110repositioned to be more central is made, and a gesture input1140is detected and identified as a de-selection gesture. The gesture1140is, for example, a short tap on the interface surface104of the computing device100remote from the connector1110. This move operation illustrates how users are able to easily adjust the rendering and repositioning of connection points made by the application112. The application112seeks to strike a balance by performing as little change to the original hand-drawn input of the user as possible while beautifying the recognized diagram so that it can be ultimately made part of a document in a typeset form that could have been created using many of the available techniques for producing digital diagrams.

InFIG. 53A, the detection of the de-selection causes the selected typeset connector1110cto be omitted and display of the connector as a digital ink connector1110a′. InFIG. 53B, the detection of the de-selection causes the selected typeset connector1110cto be omitted and display of the connector as a typeset ink connector1110b′. The newly displayed connectors1110a′ and1110b′ are the resized versions of the previously displayed digital ink connector1110aand the typeset ink connector1110b, respectively. The resizing operation of the typeset ink connector is straightforward and achieved using well-known techniques to one of ordinary skill in the art. However, the resizing operation of the digital ink is not straightforward if mere scaling or normalization of the digital ink, as is conventional, is not performed so as to avoid the resultant manipulated digital ink as being quite different to the originally drawn ink, thereby avoiding unnecessary user interaction to further manipulate the digital ink. The manner in which digital ink of a connector, and other elements, is manipulated in a resizing operation in the present system and method is now described with reference toFIGS. 73 to 77.

FIG. 73Ashows hand-drawn input of a box730and a box732connected by a bent or u-shaped connector734.FIG. 73Bshows the hand-drawn input ofFIG. 73Aafter performance of movement operations on each of the boxes730and732which cause re-sizing of the connector734, displayed as connector734′. In particular, the box730is moved to the right causing a shortening of an arm734aof the connector734connected at the box730, displayed as arm734a′ of the connector734′, and the box732is moved to the left causing a lengthening of an arm734bof the connector734connected at the box732, displayed as arm734b′ of the connector734′. The movement operations do not effect an arm734cof the connector734which is identified as connecting the arms734aand734b(and therefore the arms743a′ and743b′ of the connector734′). That is, the present system and method treats bent connectors, or other multi-arm connectors, as being formed of multiple connection portions or sub-connectors, where each sub-connector either connects to a shape, such as a container, and another sub-connector, or to two (or more) other sub-connectors. In this way, the sub-connectors can be manipulated independent of the other sub-connectors while the connections of the connector as a whole (e.g., the combined sub-connectors) are maintained. As can be seen the digital ink rendered characteristics (including ink weight and general shape) of the connector734is substantially maintained in the digital ink rendering of the re-sized connector734′. The manner of achieving this is now described with relation toFIGS. 73C to 73E, in which the arm734bof the connector734is used to illustrate the operation of the present system and method.

FIG. 73Cis a zoomed-in view of the arm734bof the connector734ofFIG. 73A(also with some portions of the box732and arm734c).FIG. 73Dis a zoomed-in view of the arm734b′ of the connector734′ ofFIG. 73B(also with the same portions of the box732and arm734c). The zoomed-in views ofFIGS. 73C and 73Dare of the same magnitude such that the zoomed-in dimensions of the arms734band734b′ are the same.FIG. 73Eis a further zoomed-in view of the arm734b, as depicted inFIG. 73C, at a magnitude so as to match the greater length of the arm734b′ ofFIG. 73D. Accordingly,FIG. 73Eillustrates scaling of the arm734bto provide the resized length of the arm734b′. As can be seen, the scaled digital ink of the arm734bis significantly different than the non-scaled digital ink, and is therefore not desirable.

Conventional normalization deals with this problem with two different techniques, which are now described in relation toFIG. 73E. In the first technique, the mean center line along the horizontal direction of the scaled arm734bis taken and the original ink weight (i.e., the average distance of the horizontal edges from the mean center line of the non-scaled arm734b) is used to regenerate the digital ink about that mean center line, thereby discarding the horizontal edges of the scaled arm734b. This operation eliminates the scaled-up rendering, but typically results in the edges of the resized digital ink element having much greater uniformity than the edges of the original digital ink, thereby significantly changing the appearance. In the second technique, the horizontal edges of the scaled arm734bare taken and the original ink weight is used to regenerate the digital ink from the edges, thereby discarding the central ink of the scaled arm734b. This operation also eliminates the scaled-up rendering, but typically results in the edges of the resized digital ink element having much less uniformity that the edges of the original digital ink, thereby significantly changing the appearance. These problems are equally applicable when the resizing operation is a reduction in size.

In the present system and method, aspects of the non-scaled and scaled versions of the digital ink are combined to provide resized digital ink elements having retained features of the original digital ink elements. That is, portions A at the ends of the arm734bare retained in non-scaled form in the resized arm734b′, as shown inFIGS. 73C and 73D. The lengths of these portions are defined so that characteristic and noticeable features, such as corners of bend connectors (as inFIG. 73), arrowheads of arrow connectors, and joints to other shapes, are maintained in the resized connector. In this way, arrowheads in particular are not resized. In the, section of the arm734bbetween the end portions A, areas of relative uniformity (i.e., gradual changes in vertical offset along the horizontal direction at the center mean line and the horizontal edges), such as the portion B inFIG. 73C, and areas of relatively high variability (i.e., sudden changes in vertical offset along the horizontal direction at the center mean line and the horizontal edges), such as the portion C inFIG. 73C, are determined.

For the areas of relative uniformity, the horizontal edges of the scaled arm are taken with ink weight adjusted to the non-resized ink weight and the central horizontal portion discarded, and rendered as portion B′ in the resized arm734b′ (as shown inFIG. 73D). Due to the relative uniformity, it can be seen that this portion of the resized digital ink is substantially similar to the original non-resized digital ink. For the areas of high variability, the center mean line of the scaled arm is taken with ink weight adjusted to the non-resized ink weight and the edges discarded, and rendered as portion C′ in the resized arm734b′ (as shown inFIG. 73D). Due to the retention of some variability, it can be seen that this portion of the resized digital ink is substantially similar to the original non-resized digital ink, particularly in the non-zoomed views ofFIGS. 73A and 73B. The digital ink of all diagram elements, both non-text and text, can be treated in the same way for resizing operations. This treatment can also be made for other operations, such as changes in aspect, direction, etc.

For example,FIGS. 74 and 75show movement operations on digital ink boxes with the consequential effect on digital ink connectors associated therewith. InFIG. 74A, a box740is selected in response to detection of a selection gesture741and moved in the direction of arrow F. The box has two associated connectors, a bent open-headed arrow connector742and straight open-headed arrow connector743. The adjusted display at the completion of the movement operation is shown inFIG. 74B, in which the connectors742and743are respectively displayed as adjusted connectors742′ and743′. The bent connector742is adjusted with the separate arms each lengthened in the manner described above. As can be seen, the dimensions of the arrowhead of the connector742are retained in the adjusted connector742′.

The connector743, which is displayed substantially vertical inFIG. 74Ais adjusted to be shortened in the manner described above and displayed at a slanted angle to the vertical as the adjusted connector743′ inFIG. 74B. As can be seen, the dimensions of the arrowhead of the connector743are retained in the adjusted connector743′. The change in angle of the connector is performed so as to retain the geometry of the connector, e.g., the adjusted connector743′ is rendered to be substantially straight like the original connector743and not caused to be curved due to the movement of box740. Such curving, for example, would be required if the connection or anchor points of the connector743to the box740and another box744were maintained for the adjusted connector743′. However, adjustment of the connection points is made to retain the connector's geometry and to provide a sensible re-display during and after the movement operation. This is achieved in the present system and method by taking account of the center of geometry of each connected shape.

As can be seen inFIG. 74A, the center of geometry of the boxes740and744, and other connected boxes745and746, are determined by the application112as indicated by the cross-marks G. The path of connection between each center of geometry and the associated connector which takes account of the geometry of the connector is also determined, shown for the connector742between the boxes740and745, for example, as dashed line747which bends at the ‘elbow’ of the bent connector742. When the box740is moved, the determined connection paths between the centers of geometry of the connected boxes are adjusted to remain between the centers of geometry whist retaining the path geometry, for example, as shown inFIG. 74B, the connection path747retains its bent geometry and connection path748of the straight connector743retains its straight geometry between the centers of gravity of the boxes740and744such that it becomes angled to the vertical. Accordingly, the adjusted bent connector742′ is rendered along the adjusted connection path747and the adjusted connector743′ is rendered along the adjusted connection path748. It is noted that the depicted marks for the centers of geometry and connection paths are provided in the drawings for illustrative purposes only, and are not typically displayed to users by the application112. However, the UI of the application112may provide users with the ability to display such markings for reference, for example during editing operations.

InFIG. 75A, a box750is selected in response to detection of a selection gesture751and moved in the direction of arrow H. The box has two associated connectors (but could have more in the present example), a first open-headed arrow connector752and a second open-headed arrow connector753, both of which have substantially straight geometry and are substantially parallel to one another. The adjusted display at the completion of the movement operation is shown inFIG. 75B, in which the connectors752and753are displayed as adjusted connectors752′ and753′. The connectors752and753, which are displayed substantially horizontal inFIG. 75Aare adjusted to be lengthened in the manner described earlier and displayed at a slanted angle to the horizontal as the adjusted connectors752′ and753′ whilst retaining the substantially parallel alignment. As can be seen, the dimensions of the arrowhead of the connectors752and753are retained in the adjusted connectors752′ and753′.

Unlike the example ofFIG. 74, a path from the centers of geometry (G) of the box750and connected box754does not flow through the parallel-aligned connectors752and753. Accordingly, the substantially parallel-alignment of the connectors is respected in the movement operation by determining the common connection path of the parallel connectors which generally extends substantially parallel to the connectors substantially centrally there between. When the box750is moved, the common connection path is adjusted to remain between the centers of geometry whist retaining the path geometry, for example, as shown inFIG. 75B, common connection path755retains its straight geometry between the centers of geometry of the boxes750and754such that it becomes angled to the horizontal. The adjusted connectors752′ and753′ are rendered along the adjusted common connection path755so as to retain the parallel separation therefrom of the original connectors752and753.

Further,FIG. 76shows a movement operation on a digital ink connector with the consequential effect on a digital ink connector label associated therewith.FIG. 76A, shows a hand-drawn diagram760rendered in digital ink. InFIG. 76B, selection of a connector761is displayed through the afore-described selection mode typeset display. In addition, the connector status of the selected connector761is indicated through display of connection points762and763at the ends of the connector761. The selected connector is sought to be moved in the direction of arrow I at the end763and during movement a connection zone764is displayed. In the example ofFIG. 76, the connection zone is displayed as a circle which is color-filled with the selection mode color which guides users during their interaction with connectors to indicate when one or more ends of the connectors are within suitable proximity (determined as described earlier) to other diagram elements to cause connection to those elements if interaction with the connectors is ceased at that point. Accordingly, if the end763of the connector761is moved too far from containers765and766display of the connection zone764is ceased until the end is brought in proximity of another element again. As can be seen, connectors can be connected to shapes and text, including sub-containers, within containers.

InFIG. 76C, the connector761has been moved so that the end763is connected to the container765at a new connection point763′. As a result, the angle of the connector761relative to the container765and a container767to which the end762is connected, is adjusted with resizing and re-display of the connector761as an adjusted connector761′, in the manner described earlier. As can be seen, this adjustment of the position and orientation of the connector761causes consequential adjustment of the position of a text label768associated with the connector761, where this association is determined in the manner described earlier. For example, the adjustment of the label768is performed so as to maintain an average distance between the text (such as from the barycenter of the recognized text) and the connector761. Accordingly, the users' labelling of the connector761is respected through the movement operation. Further, inFIG. 76C, the connector761′ is sought to be moved again in the direction of arrow J at the end762and during movement the connection zone764is again displayed.

InFIG. 76D, the connector761′ has been moved so that the end762is connected to the container767at a new connection point762′. As a result, the angle of the connector761′ relative to the containers765and767is adjusted with resizing and re-display of the connector761′ as an adjusted connector761″, in the manner described earlier. As can be seen, this adjustment of the position and orientation of the connector761′ again causes consequential adjustment of the position of the text label768. A gesture769is detected as a de-selection gesture, and accordingly the diagram760is displayed as shown inFIG. 76Ewith the moved connector761and associated text label768.

The handling of editing operations on, or which effect, complex connectors in digital ink, such as branched connectors particularly used in organizational charts (seeFIG. 4), in a manner which retains intent of users of the hand-drawn diagrams is also provided by the present system and method.FIG. 77Ashows a hand-drawn diagram770rendered in digital ink as chart having a hierarchical structure of three levels. A box771in the second level of the chart is selected in response to detection of a selection gesture772and moved in the direction of arrow K. The box has four associated connectors, a branched connector773to the first level of the chart and three ‘straight’ connectors774to the third level of the chart. The branched connector773is detected by the present system and method as being a complex or multi-connector, as described earlier, with sub-connectors formed of a trunk773adisposed in the horizontal direction with an upper branch773bextending vertically upwards from the trunk773ato a box775in the first level and three lower branches773cextending vertically downwards from the trunk773ato the box771and two boxes776in the second level. The box771and the boxes776are connected via the connectors774and connectors777, respectively, to text blocks778in the third level.

The adjusted display at the completion of the movement operation is shown inFIG. 77B, in which the connectors773and774are respectively displayed as adjusted connectors773′ and774′. As can be seen, the straight connectors774are adjusted to be lengthened in the manner described above, and the branched connector773is adjusted so that the branch773bis shortened as adjusted branch773b′, the branches773cconnected to the boxes776are lengthened as adjusted branches773c′, the branch773cconnected to the moved box771is shortened as adjusted branch773c″, and the trunk773ais retained without adjustment, albeit the relative vertical position of the trunk773abeing moved upwards to accommodate the movement of the box771there-beneath. In this way, the hierarchical structure of the hand-drawn chart is respected since the relative positions of the first and second level boxes are retained as are the relative positions of the text blocks of the third level. Alternatively, if the hierarchical structure is not determined as important, the text blocks778connected to the moved box771via the connectors774may be moved with the box771by not adjusting the connectors774, and/or the box771, and its associated connectors and text blocks, moved above the trunk of the branched connector by not adjusting the relative position of the trunk773a. Further, the trunk itself may be decreased or increased in length if the move operation causes the boxes to be further or closer to one another.

From these example editing operations on shape and text elements of the diagrams, it is understood that a large range of interactions with the digital ink elements of hand-drawn diagrams is provided by the present system and method, where the effects and outcomes of those interactions are maintained on the digital ink itself. Accordingly, users are provided with consistent control over their hand-drawn content, rather than needing to convert to the typeset version of the diagrams in order to allow editing to be performed.

Returning to the example input, inFIGS. 53A and 53B, a gesture input1141is detected as a selection gesture. The gesture1141is, for example, a long press on the interface surface104of the computing device100not on any of the diagram elements. Accordingly, the long press selection gesture is identified by the application112as initiating a free-selection mode, e.g., the free-form selection of one or more diagram elements or so-called “lasso” gesture, such that the application112expects subsequent operation performed on the selected elements. In response to this identification, a free-selection icon1142may be displayed, thereby providing the user with visual feedback of the detection.

InFIGS. 54A and 54B, a gesture input1143is detected as a selection operation in the direction of arrow C. The gesture1143is, for example, a slide on the interface surface104of the computing device100as depicted inFIG. 54. In response to this detection, a selection zone1144is displayed with selection mode rendering, thereby providing the user with visual feedback of the detection.

InFIGS. 55A and 55B, continuation of the free-selection gesture1143is detected and recognized as extended display of the selection zone1144.

InFIGS. 56A and 56B, completion of the selection operation is detected due to the free-selection gesture1143completing a roughly circular form about certain elements of the diagram1100and a selection box1145is displayed as a result about the diagram elements within the completed selection zone1144, namely the respective digital and typeset ink forms of the duplicate text1106,1109,1112and1113, the text1116and1124, the duplicate shape1103, and the shapes1110,1114,1117,1119,1122,1125and1129. These selected elements are displayed as selected typeset ink1103c,1106c,1109c,1110c,1112c,1113c,1114c,1116c,1117c,1119c,1122c,1124c,1125cand1129c, having the selection mode properties, and the non-selected elements are displayed having the non-selection properties of the multi-selection mode, thereby providing the user with visual feedback of the detection. Accordingly, multiple recognized diagram elements can be quickly selected.

In the example shown inFIG. 56, the free-form selection mode is configured so that both elements that are completely enclosed by the selection zone and also only partially within the selection zone, i.e., the connector arrow1114, are recognized as selected. Alternatively, the free-form selection mode may be configured so that only those elements that are completely enclosed by the selection zone are recognized as being selected. For the latter case, selection of elements that are partially enclosed for inclusion in the multi-selection can be made by users through selection gestures, such as those described earlier. In any case, in the free-form and multi selection modes, selected elements can be de-selected through the same gestures. For example, inFIGS. 56A and 56Ba gesture input1146is detected and identified as de-selection of the selected typeset connector1114c. The gesture1146is, for example, a short tap on the interface surface104of the computing device100on the connector1114and within the selection box1145, as depicted inFIG. 56.

InFIGS. 57A and 57B, the detection result causes the selection mode typeset display of the de-selected connector1114to be omitted with display reverting to the respective normal digital and typeset ink and consequential contraction of the selection box1145to no longer surround the de-selected connector1114. Further, a gesture input1147is detected as selection of the selection box1145. The gesture1147is, for example, a long tap on the interface surface104of the computing device100within the selection box1145but not on any of the diagram elements therein, as depicted inFIG. 57. In response to this recognized gesture, the application112displays the selection menu1134. A further gesture input1148is detected as selection of the Duplicate operation from the menu1134causing the selected elements to be duplicated. The duplicate elements are retained in the selection box1145in the selected mode rendering with the same positional relationships as the original elements that have been duplicated, however the display of the elements that have been duplicated returns to the normal rendering or reverts to the non-selected rendering, with reduced emphasis, like inFIG. 57. The gesture1148is, for example, a press on the interface surface104of the computing device100as depicted inFIG. 57.

InFIGS. 58A and 58B, a gesture input1149is detected and identified as a move operation on the selection box1145in the direction of arrow D. The gesture1149is, for example, a long tap-and-slide on the interface surface104of the computing device100within the selection box1145but not on any of the diagram elements therein, as depicted inFIG. 58.

InFIGS. 59A and 59B, the gesture input1149is continued to be detected and identified as moving display of the selection box1145and the duplicate elements contained therein together with display of alignment guides1150as dashed vertical lines along the left and right boundaries of the selection box1145and extending there-above and -below. The use, rendering and timing of display of such alignment elements is understood by one of ordinary skill in the art, and may be made with respect to an alignment grid underlying the display interface, which grid may also be used to define one or more extents of the selection box, as shown inFIG. 59. The alignment guides1150are provided to guide users in their placement of the moving elements with respect to the non-moving elements of the diagram1100. For example, inFIG. 59the move operation on the selection box1145by the sliding gesture input1149is recognized in the direction of arrow E whereby the user is seeking to position the duplicated elements with respect to the non-selected elements. At completion of the move operation a gesture input1151is detected as a de-selection gesture. The gesture1151is, for example, a short tap on the interface surface104of the computing device100outside of the selection box, as shown inFIG. 59.

InFIGS. 60A and 60B, in response to completion of the move operation display of the selection box1145is omitted and the selection mode typeset display of the de-selected diagram elements is omitted with display reverting to the respective normal digital and typeset ink, and the display of the other non-selected diagram elements is returned to the respective normal digital and typeset ink. Due to the completed move operation display is made of the duplicate container1117apositioned adjacent the original arrow1122inFIG. 60Aand the duplicate typeset ink container1117bpositioned adjacent the original typeset arrow1122binFIG. 60B. In this process, the connector status of the original arrow1122is affirmed due to its connection between the original container1119and the duplicate container1117.

Further, inFIG. 60A, a hand-drawn shape1152input over the open-arrowhead of the duplicate arrow1122ais displayed as a digital ink shape1152a. InFIG. 60B, the hand-drawn shape1152input over the open-arrowhead of the duplicate typeset arrow1122bis displayed as the digital ink shape1152a. The shape1152is hand-drawn in a single continuous stroke, detected as non-text and identified as a closed arrowhead due to the characteristics of the shape1152. As described earlier, this detection of a shape substantially overlaying a previously recognized shape is interpreted by the application112as an overwrite gesture. Accordingly, the overwritten shape is deleted or omitted from display and replaced with the new shape.

InFIG. 61A, the overwrite causes the open-headed duplicate arrow1122ato be removed from display and display of a closed-headed arrow1153as a digital ink closed-headed arrow1153a, with the point of the hand-drawn arrowhead1152repositioned on the end of the ‘stem’ of the duplicate arrow1122a, thereby substantially reforming the open-headed duplicate arrow1122aas a closed-headed arrow. InFIG. 61B, the overwrite causes the typeset open-headed duplicate arrow1122bto be removed from display and display of the closed-headed arrow1153as a typeset closed-headed arrow1153b, with the point of the hand-drawn arrowhead1152repositioned on the end of the ‘stem’ of the duplicate typeset arrow1122b, thereby substantially reforming the duplicate open-headed typeset arrow1122bas a closed-headed arrow.

Further, inFIG. 61A, a hand-drawn input1154is displayed as digital ink1154aoverlaying the duplicate words1116a. InFIG. 61B, the hand-drawn input1154is displayed as digital ink1154aoverlaying the duplicate typeset words1116b. The input1154is detected as a handwritten editing gesture and identified as a scratch-out (erase mark) on the words1116contained within the duplicate container1117. This scratch-out editing operation is interpreted by the application112as an erasure of the words1116.

InFIG. 62A, the identified editing operation causes the duplicate words1116ato be removed from display such that the duplicate container1117ais empty. InFIG. 62B, the identified editing operation causes the duplicate typeset words1116bto be removed from display such that the duplicate typeset container1117bis empty

InFIG. 63A, handwritten text1155input within the duplicate oval1117ais displayed as digital ink text1155a. InFIG. 63B, the handwritten text1155input within the duplicate typeset oval1117bis displayed as the digital ink text1155awithin the duplicate typeset oval1117b. The text1155is detected as text and recognized as the phrase “Planning & Spec”, and the duplicate oval1117is affirmed as a container which contains the recognized phrase1155due to the relative positions and characteristics of the elements1117and1155. InFIG. 63B, it can be seen that the handwritten text1155ais not fully contained in the typeset container1117bdue to the typeset size of the container. The application112of the present system and method detects that the text1155is intended as contained text or a container label, for example, despite some of the handwritten input overlapping the boundaries of the container. This situation may occur also where text or shapes are, or desired to be, written to not completely fit within a digital ink container. These situations are handled however as described in the examples ofFIGS. 78 to 83.

FIG. 78illustrates an example in which text has been handwritten within a container but does not fit within that box. InFIG. 78A, the handwritten input of a shape780and text781is displayed as a digital ink box780ahaving digital ink text781a mostly contained therein. The text781is detected as text and recognized as the phrase “This is a too long text”. The box780is detected as non-text and recognized as a rectangle. In order to identify the rectangle780as a container containing the recognized phrase781, the relative positions and characteristics of the inputs780and781are considered. For example, a pre-set, and settable, spatial threshold is used by the application112where comparative metrics of geometric features of the rectangle780and the text block781relative to that threshold indicate the likelihood of the text block being contained in the rectangle. For example, the distance between the mean centers or barycenters (as described above) of the rectangle780and the text block781is compared to a distance threshold (as described earlier), or the proportion of overlap of the areas of the rectangle780and the text block781is compared to an overlap threshold where a high likelihood of containment is indicated when more of the text is written within the potential container than without.

The determination of container status is one consideration and provides reliable and intuitive association of the different diagram elements. Another consideration is the rendered display of the hand-drawn diagram. As explained earlier, the present system and method seeks to perform the minimum amount of automatic manipulation of users' input as possible. However, legibility and sensible display of the input in both digital and typeset ink is supported by implementing a certain degree of refinement, so-called beautification. InFIG. 63, it was seen that a problem occurred with the input of the text1155over the typeset container1117bnot the digital ink container1117a, due to the typesetting employed by the application112during incremental typeset which (slightly) shrinks the digital ink footprint, e.g., the typesetting or fontification results in typeset ink of (slightly) smaller dimensions than the digital ink. A similar issue can occur however if the typesetting enlarges the digital ink footprint, e.g., the typesetting results in typeset ink of (slightly) larger dimensions than the digital ink. The effects of typesetting in this way may be beneficial in other situations however. For example, inFIG. 78B, the recognition result ofFIG. 78Ais displayed as a typeset phrase781bcompletely contained within a typeset container780b, unlike the digital ink versions inFIG. 78A.

Beautification can also be carried out on the typeset ink. For example, inFIG. 79, the handwritten input of a shape790, text791and additional shapes792and793is displayed as a digital ink box790ahaving digital ink text791a mostly contained therein, and a digital ink box792awith a digital ink arrow793abetween the boxes790and792. The text791is detected as text and recognized as the phrase “This is a way too too too too too long text”. The boxes790and792and arrow793are detected as non-text and recognized as rectangles and an arrow, respectively. Further, the box790is identified as a container790containing the recognized phrase791, and the arrow793is identified as a connector connecting the container790and the rectangle792. The application112however further detects that due to the size of the text block of the text791compared to the size of the container790, e.g., through comparison of the respective geometrical areas, that upon typesetting the typeset text will not fit within the typeset container, which would lead to undesirable rendering. This can be handled, for example, by reducing the size of or changing the type of the font of the typeset text, however this may result in text in different parts of a diagram being of different font size or type (e.g., the present system and method may implement a default, user settable, font style for text) which is undesirable.FIGS. 80 and 81illustrate different mechanisms for handling this situation automatically, andFIGS. 82 and 83illustrate manual mechanisms.

InFIG. 80, the vertical extent (i.e., height) of the container790is extended by the application112along arrows L to be displayed as a typeset container790bcontaining display of the phrase791as a typeset phrase791bcompletely therein. This is achieved by the application112determining the required extent of typeset container to contain the typesetted dimensions of the text. As can be seen, the display of the rectangle792and connector793as a typeset rectangle792band connector793bis consequentially moved downwards to accommodate the enlarged container790b. This beautification of the typeset ink therefore provides clean display of the diagram elements whilst retaining the relationships.

Alternatively, inFIG. 81, the container790is displayed as a typeset container790b′ without the extension ofFIG. 80, and rather the phrase791is displayed as a typeset phrase791b′ completely within the non-resized container790by reflowing the typeset ink as shown by arrow M. This is achieved by the application112determining the relative extent of such reflowed text to the typesetted dimensions of the container. Accordingly, the display of the typeset rectangle792band connector793bis not moved. This beautification of the typeset ink therefore provides clean display of the diagram elements whilst retaining the absolute positions.

Alternatively, or in addition to, the above-described automatic beautification through resizing and reflowing of the typeset ink, the ability to manually beautify input hand-drawn diagrams is also provided by the present system and method. InFIG. 82A, the container1117containing the text1116is shown in isolation from the diagram1100as displayed in selection mode of the container1117, that is as a selected typeset container1117ccontaining the digital ink word1116aand having a selection box820thereabout. InFIG. 82B, a gesture input821on a resize handle820aof the selection box820is detected as a move operation on that resize handle820ain the direction of arrow N. This operation causes resizing of the display of the selected typeset container1117cas resized selected typeset container1117c′ as depicted inFIG. 82B. However, as can be seen, the non-selected text1116is not moved during the resizing of the container1117. This is because the application112defines containers in such a way that the container and its contained contents can be treated independently whilst retaining the container/contained relationship.

InFIG. 82C, the detection of a de-selection gesture822causes display of the container1117as a digital ink container1117a′ which is a resized version of the digital ink container1117a. InFIG. 82D, handwritten text823input within the container1117a′ below the text1116ais displayed as digital ink text823a. The application112detects that the text1116and823are related due to the relative positions and characteristics of the inputs1116,1117and823(for example, the application112uses a pre-set, and settable, spatial threshold where the proximity of the text1116and823to each other less than the threshold provides a high likelihood of the text belonging to a text block).

Alternatively, inFIG. 83A, the container780and text781ofFIG. 78Ais shown as displayed in selection mode, that is as a selected typeset rectangle780ccontaining a selected typeset phrase781c. As the container780cis a rectangle it is displayed with selection box handles830. Further, a gesture input831on one of the resize handles830is detected as a move operation on that resize handle830in the direction of arrow O.

InFIG. 83B, the resizing operation causes resizing of the container and its content and display of a selected typeset rectangle780c′ and selected typeset phrase781c′, which is a reflow text block of the text781. Accordingly, as both the container and the text were selected, the text is moved during the resizing of the container. InFIG. 83C, the de-selected resized elements are displayed as a digital ink container780a′ and digital ink text781a′.

In the example ofFIG. 82handwritten text was added to a container containing digital ink text after resizing of the container. The ability of adding text to a container can however also be performed after typesetting of already contained text, with or without resizing of the container. This can be done in already created diagrams that have been typeset or during incremental typesetting at diagram creation as depicted for the example diagram1100. An example of the incremental typesetting is illustrated inFIG. 84. InFIG. 84A, handwritten text840input within a shape841is displayed as digital ink text840aand a digital ink shape841a, respectively. The text840is detected as text and recognized as the words “This is” and the shape841is detected as non-text and recognized as a rectangle. InFIG. 84B, the recognition result is displayed as a typeset rectangle841bcontaining typeset words840b, with the relative positions and dimensions substantially maintained. InFIG. 84C, handwritten text842input within the typeset rectangle841bto the right of the typeset words840bis displayed as digital ink text842a. The additional text842is detected as text and recognized as the words “a cell”. InFIG. 84D, the recognition result is displayed as typeset words842b, with the relative positions and dimensions of all the elements substantially maintained. As in the example ofFIG. 82, the application112detects that the text840and842are related due to the relative positions and characteristics of the inputs840,841and842, and therefore groups them as text843and causes the combined text840and842to be recognized as a phrase “This is a cell”.

InFIG. 84, the typeset text within the container is retained in the same relative position within the container as the digital ink text, namely on the left-hand side or left-justified, despite the availability of space on the right of the text within the container, and therefore respects the user's handwritten input. Alternatively, the present system and method may cause the text position to be adjusted when there is space within the container, such as, centering the text as shown inFIG. 78B, for example. This beautification can be done for either or both typeset and digital ink text, and also for contained shapes, and is provided as a setting for user selection, for example. Generally, such beautification is performed differently for containers and cells, as defined earlier. In particular, for cells which only contain text, the text within the cell may be treated as a label of the cell. Accordingly, the input text is beautified in both digital and typeset ink to be centered (both horizontally and vertically) within the cell, for example. This is illustrated inFIG. 85A, in which a digital ink label850is centered in digital ink cell851. The cell851is shown in selection mode and displayed as resized inFIG. 85Bin which the digital ink label850remains displayed at the center as is defined for cell labels, for example.

Alternatively, such resizing of a cell (e.g., beyond a pre-set, and settable, dimension threshold) may cause the cell to lose its definition as a cell, in which case the contained text is no longer defined as a label in the resized container and is treated as a text block rather than a label, with or without associated beautification. This allows other text blocks and shapes to be easily input into the resized container. Alternatively, or in addition, the loss of cell definition may occur when additional text or shapes are hand-drawn within the resized (or originally drawn sized) cell in a manner in which the application112does not automatically link the text to the (beautified) text label (e.g., the added text is without the afore-described threshold used for relating text). The opposite conditions may be used to re-classify containers as cells.

FIG. 86illustrates an example of plural text blocks within a container. InFIG. 86A, an input shape860with text861and862therein is displayed as a digital ink container860acontaining digital ink text861aand digital ink text862a. Upon detection of a selection gesture863the text block862is displayed in selection mode as selected typeset text block862′ inFIG. 86B. This illustrates that the text861and862are identified as being individual text blocks in the container. InFIG. 86C, the input860,861and862have been typeset and are displayed as a typeset container860band typeset text blocks861band862b.

Returning to the example input, inFIG. 64B, the recognition result is displayed as a typeset phrase1155band the duplicate typeset container1117bautomatically resized as a container1156containing the typeset phrase1155bdisplayed as a typeset oval container1156b, with the automatic resizing carried out as discussed above in relation toFIG. 80. Further, inFIG. 64A, a hand-drawn input1157is displayed as digital ink1157aoverlaying a part of the duplicate term1113a. InFIG. 64B, the hand-drawn input1157is displayed as the digital ink1157aoverlaying a part of the duplicate typeset term1113b. The input1157is detected as a handwritten editing gesture and identified as a scratch-out on part of the duplicate label1113on the duplicate connector1103. This scratch-out editing operation is interpreted by the application112as an erasure of the character “S” in “S-Gate” of the duplicate term1113.

InFIGS. 65A and 65B, the erase operation causes the duplicate term1113aand typeset term1113b, respectively, to be removed from display and display of a term1158respectively as a digital ink term1158aand a typeset term1158b, namely “-Gate”.

InFIG. 66, handwritten text1159input to the left of the text1158ais displayed as digital ink text1159a. InFIG. 66B, the handwritten text1159input to the left of the typeset text1158bis displayed as the digital ink text1159a. The text1159is detected as text and recognized as the character “I”. The text1159is recognized as being associated with the text of the label1157due to their relative positions and characteristics, e.g., both are text objects, such that the term “I-Gate” is recognized.

InFIG. 67, the recognition result causes the typeset term1158bto be removed from display and display of a term1160as typeset term1160b, namely “I-Gate”. Upon the detection of further minor inputs similar in nature to those already described, the fully typeset version of the diagram1100shown inFIG. 11Bis provided by the application112.

The application112provided by the present system and method faithfully allows display of handwritten diagrams in both digital ink and typeset ink forms. The hand-drawn elements are identified, their content recognized using handwriting recognition, and their spatial and context relationships detected, independent of the type of diagram created and the order of drawing of the elements. The diagrams can be edited in both digital ink and typeset ink forms using handwritten and computing device gestures.

While the foregoing has described what is considered to be the best mode and/or other examples, it is understood that various modifications may be made therein and that the subject matter disclosed herein may be implemented in various forms and examples, and that they may be applied in numerous other applications, combinations, and environments, only some of which have been described herein. Those of ordinary skill in that art will recognize that the disclosed aspects may be altered or amended without departing from the true spirit and scope of the subject matter. Therefore, the subject matter is not limited to the specific details, exhibits, and illustrated examples in this description. It is intended to protect any and all modifications and variations that fall within the true scope of the advantageous concepts disclosed herein.