A Generic Approach to the Recognition and Analysis of Sketched Diagrams Using Context Information

Recent decades have shown the rise of diagrammatic representation of information. In computer sciences, for example, general purpose diagrammatic notations like the UML are an everyday tool nowadays, and can even be considered as common knowledge. Also the advent of domain-specific languages (DSLs) can be observed. On the other hand, the research field of sketching is becoming popular, due to advances in processing speed and input hardware. Also, fields of application are evident, e.g. drawing of diagrams. The term sketching means to have a user draw something and have the computer interpret the drawing in some appropriate way. The advantage of sketching over traditional WIMP-based user interfaces (window, icon, menu, pointing device) is a more natural and intuitive way of interaction with the computer. This thesis presents DSKETCH, an approach to sketching of diagrams. The idea is that the user first draws a diagram, and then DSKETCH derives the syntactic and semantic information conveyed in the drawing. The semantic information can be used for subsequent processing. The approach is fully generic, i.e., it is not tailored to a specific diagram language. There is a prototypically implemented system which serves as proof-of-concept. As an example, the user draws a class diagram from the UML. The system then derives the semantics of the diagram, and creates skeleton class files. The user can subsequently create an actual implementation with these skeletons. Reaching this goal depends on two subsequent stages, applied after the user is finished drawing. The first is recognition, which means to identify the single shapes that make the complete diagram. The other step is analysis, which means to inspect each shape in the context of other shapes, thus being able to derive a syntactical structure first, and the semantics afterward. Recognition is subject to much current research in the field of sketching. Still, no satisfying solution could be found yet. State-of-the-art approaches mostly constrain the user and impose restrictions regarding how to draw. Thus, the task of recognition is simplified to a point where it becomes bearable, but the user is forced to concentrate on his drawing style. Analysis, on the other hand, is rarely discussed in publications on sketching. However, analysis should be an important aspect of approaches to iii iv sketching, as the user is usually not interested in recognized shapes, but in the semantics of the diagram. The approach presented in this thesis marks improvements, both for recognition and for analysis. The core idea of the recognition is to avoid a featurebased approach for high-level recognition, as features impose severe restrictions on which drawings can be recognized. Instead, a set of independent models is created, all of which contain information gained from low-level processing. Furthermore, multiple representations of the same stroke in different models are possible. This has a positive effect on recognition, because it removes the task of low-level processing to decide for a suitable representation without any context knowledge. High-level recognition itself is then based on composition of primitives to complete shapes. In general, the presented approach to recognition does not constrain the user in the ways shown by previous work in the field. Analysis builds upon the DIAGEN framework, which allows for generation of WIMP-based diagram editors from specifications. The generated editors allow for checking syntax and semantics of the diagrams created by the user. The concept of DIAGEN is based on the formal approach of graph transformation, which results in a powerful and reliable diagram analysis. In this thesis it is shown how this approach can be transferred to sketching. The most distinct result is that ambiguities can be reliably solved by extensive use of context information gained from syntax checking. Ambiguities naturally arise from hand-drawing, which is inevitably sloppy and imprecise. Furthermore, it has proven valuable to explicitly model ambiguities in the analysis process. Also, diagram language-specific output can be generated as a result from the analysis as motivated above with the example of class diagrams. The prototypical implementation is applied to six different diagram languages, all of which exhibit different characteristics regarding visual appearance, syntax, and semantics. Among the six languages there are statecharts from the UML, and a GUI builder as a representative of DSLs. Many further diagram languages are conceivable as well. An empirical user study evaluates recognition rates and performance of the prototype. It proves that the system is both accurate and powerful. The contribution of this thesis lies both in the recognizer and the analysis. The recognizer allows for multiple representations of the same stroke at the same time, and is capable of identifying shapes from a complete drawing without prior assignment of strokes to shapes. The analysis is based on a formal approach. Ambiguities are solved automatically based on the syntactic structure of the diagram language. Therefore, ambiguities are explicitly modeled for the analysis.