Automating object transformations for dynamic software updating Dynamic software updating (DSU) systems eliminate costly downtime by dynamically fixing bugs and adding features to executing programs. Given a static code patch, most DSU systems construct runtime code changes automatically. However, a dynamic update must also specify how to change the running program's execution state, e.g., the stack and heap, to make it compatible with the new code. Constructing such state transformations correctly and automatically remains an open problem. This paper presents a solution called Targeted Object Synthesis (TOS). TOS first executes the same tests on the old and new program versions separately, observing the program heap state at a few corresponding points. Given two corresponding heap states, TOS matches objects in the two versions using key fields that uniquely identify objects and correlate old and new-version objects. Given example object pairs, TOS then synthesizes the simplest-possible function that transforms an old-version object to its new-version counterpart. We show that TOS is effective on updates to four open-source server programs for which it generates non-trivial transformation functions that use conditionals, operate on collections, and fix memory leaks. These transformations help programmers understand their changes and apply dynamic software updates. Suppose you are running an on-line service and a memory leak in your server software causes it to regularly run out of memory and crash. Eventually you discover the one-line fix: the Connection class's close method should unlink some metadata when a connection closes. To apply this fix in a standard deployment, you stop your server and restart the patched version, but this disrupts active users. With dynamic software updating (DSU) support in an extended Virtual Machine such as LiveRebel [16] you can do better. You apply a dynamic patch to the Connection class of your running system to prevent further leaks without disrupting current users. In some DSU-enhanced VMs, such as Jvolve [14] , you can do better still. You include state transformation code in the dynamic patch that traverses the heap and unlinks the useless metadata left reachable by the bug.. An important goal toward furthering the adoption of DSU systems is to make them easy to use, i.e., to minimize the effort required to produce a correct dynamic patch from two versions of a system. As a step in this direction, many DSU systems employ simple syntactic, type-based tool support for constructing a dynamic patch from the old and new program versions [1, 8, 13, 14] . For example, if the bytecode for method m of class C changes, Jvolve will include C.m in the dynamic patch. If C's field definitions change, in type or number, Jvolve creates a default object transformation function that it applies to all C objects when it applies the patch. This function retains the values of unchanged fields and initializes the rest with a default value, e.g., null.. While tool support for identifying changed code is highly effective, existing support for constructing state transformation code is rarely sufficient, and the programmer must therefore modify the generated code. For example, in Jvolve the programmer must add code that unlinks the leaked metadata in our example. Unfortunately, the cases that require manual intervention are often challenging to get right. For the above example, the Connection transformer cannot simply unlink all connection metadata unconditionally. Instead, it must use appropriate context by examining the running program's heap and stack to identify and unlink only the metadata that is logically dead. Transformations that move objects between collections or partition single objects into several objects-examples we observe in practice-require similar care in their construction. Thus, writing state transformation code for DSU systems is a programming task unique to DSU, and it can be a time-consuming, error-prone process.. To ease this burden on programmers, we have developed a general-purpose approach for synthesizing object transformers that we call Targeted Object Synthesis (TOS). Our development is in the context of a DSU system for Java, called Jvolve, but our techniques are readily adaptable to other DSU systems. Furthermore, the techniques that we design for finding heap object correlations between different program versions may be useful for other program understanding tasks, such as bug detection and testing.. TOS works in two phases, matching and synthesis-the matching phase creates examples by pairing objects in two snapshots taken at equivalent points during the execution of the old and new programs, respectively, while the synthesis phase generates a function that transforms the old object of a matched example pair to the new object.. The matching phase begins by running both the old and new versions of the program on the same inputs and taking heap snapshots at corresponding program points. Given a class C whose fields changed between versions, we first reduce the heap snapshots such that they only include C objects and objects to which they refer, directly or transitively. TOS matching seeks to correlate objects in the old and new versions. TOS identifies key fields in the objects that (1) uniquely identify the object in a given heap (i.e., each object of class C differs on the values of its key fields) and (2) there exist objects in both heaps with the same values for these fields. We use a greedy algorithm which, in our experience, usually succeeds in finding a set of key fields. In the case that no key fields exist, matching uses the most distinguishing set of fields it can find to pair up most objects, and then applies a lightweight form of synthesis to find a function that pairs up the remaining objects.. With example pairs of corresponding old o and new o objects (o, o ) in hand, the synthesis phase searches for functions that are consistent with the examples, i.e., functions δ for which δ (o) = o for all matched pairs. Functions δ assign an expression to each new-version object field one at a time, where the expressions may reference any of the old object's fields (or fields reachable from them). These expressions may contain constants, simple functions (e.g., the concatenation or partitioning of string expressions), and conditionals (e.g., if the value to assign to a field depends on the current value of another field). While these expression forms are sufficient for our examples, additional expression forms can be readily supported, expanding expressiveness at the cost of increasing the search space. For collections, we recursively invoke synthesis to generate transformations between objects that make up each collection, mapping the resulting function over the old collection objects to produce the new one. When many functions are possible for a given set of examples, synthesis chooses the simplest. We carefully designed the transformation language to make important operations efficient, such as intersecting a set of candidate functions.. As far as we are aware, the TOS matching algorithm is new. No prior work attempts to map heap objects from unstructured heap snapshots of different program-version executions. The TOS synthesis algorithm is inspired by recent work on synthesizing string and Excel table data transformation functions from input and output examples [5, 6] . TOS matching creates examples automatically, whereas this prior work requires users to provide examples. TOS functions are a superset of string transformations. Whereas Excel table functions focus on filters and numerics, TOS data transformations focus on a more general problem, the transformation of heap objects.. We demonstrate our approach by synthesizing transformation functions for updates to several open-source Java servers, including JavaEmailServer (a POP and SMTP server), CrossFTP (an FTP server), Azureus (a Bittorrent client), and JEdit (a graphical text editor). In one case, changed objects do not have the key fields that matching requires. In all the others, we show that TOS produces correct transformation functions. These functions include object field additions, string partitioning, partitioning a collection based on a predicate, and deleting objects due to memory leaks. In fact, to our knowledge no prior DSU system has considered the need to correct the residual effects of bugs, such as a memory leak, and our synthesized functions are the first demonstration of this capability. TOS represents a substantial step toward realizing the promise of DSU technology by reliably automating the most programmer-intensive step. This section presents an overview of synthesizing state transformation functions for dynamic software updates using TOS. We begin with some background on DSU, present an example dynamic update taken from an actual program change, and show how TOS synthesizes this dynamic update automatically. To use a typical DSU system, we must construct a dynamic patch [8] that specifies the changed code and a state transformation function, which modifies heap objects and other program state, as necessary, to work with the new code. For example, if the old program version maintains a list of Connection objects and the new version adds some fields to the Connection class, the state transformation function must initialize the values of the new fields for the existing objects. In some systems, the state transformer may also update the control state of the program, e.g., examining and modifying the existing stack and program counter as necessary [10, 14] . We implement TOS for Jvolve [14] , which performs DSU in a Java Virtual Machine (Jikes RVM), but the TOS design generalizes to other DSU systems. Figure 1 depicts a dynamically updated program's execution. The circles represent the program's state, the labels a 1 , a 2 , a 3 , a 4 represent actions, e.g., messages sent to and from client applications. Each gray circle represents a state in which a dynamic update is permitted-not every program state may be amenable to certain dynamic updates, as discussed below. In this trace, the program starts executing at version 1, and after executing actions a 1 and a 2 , it applies a dynamic patch. As a result, the code of the program is updated to version 2, and the state transformation function δ is applied to transform the current state. A patch could have been applied in the initial state or the one after a 3 , if a patch were available at those times. This paper has presented Targeted Object Synthesis (TOS), a novel technique that synthesizes object transformer methods. Object transformers convert old version objects to new ones during a dynamic software update. TOS is distinguished by its generality: whereas prior techniques for syn-thesizing object transformers follow simple syntactic rules, TOS produces functions based on observations of actual program executions of the old and new program versions. In particular, TOS takes periodic heap snapshots at corresponding points during executions of the old and new program when executing the same inputs. It then matches corresponding objects between these snapshots, and uses these as examples to synthesize object transformation functions. We show TOS is efficacious in synthesizing transformation functions for actual changes to classes in various Java server applications. Even when it fails to generate a correct transformer, the partial results may be useful to developers. This functionality eases, but does not eliminate, the programmer burden of understanding program changes and performing dynamic software updating. TOS may also be useful for other version and program understanding scenarios, such as bug detection and testing. A key difference between TOS and prior work on learning from examples is that in prior work, the user identifies particular example pairs, whereas in our work, the user must produce matching executions-created by running the old and new program versions on the same inputs-from which TOS automatically identifies examples. While automatic matching is simpler than identifying example object pairs directly, there is a risk that it will not pair up truly corresponding objects, in which case the synthesized transformation function will be incorrect. To reduce the likelihood of this case, we place strong restrictions on the inputs to matching, as described in the third paragraph of Section 2.4: snapshots in both versions must be taken at the same update points in both versions, there must be an equal number of snapshots, and when synthesizing a transformer for class C, there must be an equal number of C objects in corresponding snapshots. These restrictions ensure that C objects are playing the same role in the old and new execution, so if we update the old program at one of these update points, executing the synthesized transformer would bring the program to an equivalent state.. Note that while corresponding snapshots must contain the same number of C objects, where C is the changed class, the number of other objects can vary. For example, a buggy implementation of C may fail to null its field f of class D, inducing a memory leak. Thus in the old and new snapshots, the number of C objects will be the same, but the number of D objects may differ. As we show in our experiments, inferring the transformer method for such a leaky class C can end up correcting the memory leak by nulling the dead objects.. Because TOS generates a solution specific to the examples it is given, the developer must snapshot executions that produce a sufficient number of objects. For conditional transformers, the input must produce objects that cover the range of possible variations; i.e., if there are N conditions, TOS needs at least an example pair for each of the N conditions. These examples could come from a single snapshot with N instances of the object in both the old and new version or from multiple snapshots which each contain one or two instances but together provide N examples. If there are too few examples, then synthesis may infer a function that is overly specific.. In our experience (described in Section 5), it was easy to provide enough examples for synthesis, and most experiments required just one (well-chosen) test. TOS fails in two of our test cases. In one case we could not reproduce a memory leak we were aiming to fix, leading to a failure during the snapshot collection phase. In the other case, the changed objects did not have key fields and so the matching phase failed. Despite these limitations, our approach adds value: the developer needs to run tests anyway, and if these tests are sufficiently deterministic and cover the relevant behaviors of a changed class, TOS can be used to infer object transformers for that class.. Our synthesis algorithm is engineered to favor simpler, more general transformations δ over more specialized ones. For example, suppose class Foo contains an integer-valued field f and in all our snapshots σ ∈ σ old we have two and not the function. Line 10 of synthesize (Figure 8 ) favors initializers that apply to the most possible objects. Line 16 of synth conj ( Figure 10 ) similarly aims for simpler, more general conditions. The algorithm as described assumes that synthesis takes place on a per-object basis, one field at a time. In fact, it can also synthesize the contents of a new object's children, e.g., assigning not just n. f := d but n. f 1 . f 2 := d as well. Likewise it can read children of old objects in initializers d. To support this extension, we simply consider field paths rather than fields, up to a specified depth, e.g., on line 2 in Figure 8 and on line 11 in Figure 9 . Matching can be similarly extended, e.g., using paths on line 12 in Figure 6 . Finally, we extend the synthesis language in Figure 7 to allow object allocation, so we can allocate and initialize child objects if needed. (Note that allocation happens implicitly with map when producing the new collection.) We have found this flexibility useful in practice, as we discuss in the Azureus example in the next section. In our experiments, we consider two updates to Azureus (a bittorrent client); three updates to jEdit (a text editor); one update to JavaEmailServer (a.k.a., JES, an SMTP and POP mail server); and one update to CrossFTP (an FTP server). These applications range from 2.3k to 250k source lines of code. These are actively developed and maintained programs not built with dynamic software updating in mind.. We chose these applications and updates for the following reasons. First, dynamic updates for these programs would be useful. For Azureus, JES, and CrossFTP, dynamic updates would avoid disruption due to shutdown/restart, e.g., they would preserve active connections involving e-mail sending or file transfer, and they would preserve important inmemory state, such as file/metadata caches. Dynamic updates to JEdit are less important (you could save in-progress work and restart), but would be convenient, e.g., to preserve the content and layout of onscreen windows. Second, the updates to JES and CrossFTP were also considered in the original Jvolve work [14] . We wanted to see if we could use TOS to generate transformers we previously wrote by hand. Finally, we found updates to these programs that are relatively interesting, such as updates that do not affect all objects uniformly (thus requiring conditional transformers) or updates that fix memory leaks. We focus on the challenging updates that prior DSU systems cannot handle automatically. We also tested TOS on many simple updates, and include two of these-the "copy" transformers for CrossFTP v1.06-as exemplars. gram, the number of object instances present (total across all snapshots), and the total number of instances of the target object (target objects changed or transitively referred to by changed objects). Finally, we list execution times in seconds for matching and synthesis, and indicate if synthesis succeeded.. Matching and synthesis times are negligible for all examples. Matching times and effectiveness benefit from focusing on a single class at a time, since it significantly reduces the number of objects that TOS considers. TOS restricts itself to changed objects and objects reachable from changed objects by a bounded number of field dereferences. This bound is the same one mentioned in Section 4.5, which discusses bounding the depth of field paths. Synthesis benefits from the fact that our language of state transformers is designed to be small and succinct.. The synthesized functions involve constant updates, conditional updates, string transformations, and collection updates. For JES and CrossFTP, we used the generated update functions in Jvolve [14] and verified that the system continued running correctly following the update. We were unable to get Azureus and jEdit to run reliably using the Jvolve VM, but this was not due to Jvolve: the Jikes RVM, on which Jvolve is based, does not run them properly either. Since we could not run the Azureus and jEdit updates, we performed a manual code review of the transformers produced by our synthesis algorithm to check that they matched what we would have written by hand.. TOS fails in two cases to synthesize update functions because we could not generate snapshots that captured the changed behavior; we discuss these situations, and all of the updates, in detail below.. Azureus update SVN r2514 Azureus is a widely used BitTorrent server/client. Version r2514 contains about 250k lines of code. This update changes two classes and methods. Figure 11 shows a portion of the update. The added call to clearServerAdapter nulls the adapter field of the server object when a peer server is stopped; doing so ensures the adapter is garbage collected. When we apply this update at run-time, we would like it to retroactively null adapter fields that the buggy version neglected to. TOS facilities doing so by synthesizing a transformer for PEPeerControlImpl objects.. To generate the snapshots, we ran both program versions in a controlled setting and had them download the same set of files. TOS performs matching on PEPeerControlImpl objects. The program contains one such object for each file it downloads. Matching chooses key field nbPieces, which is a value proportional to the file size that is highly likely to be unique across different files.. Synthesis then observes that for these matching objects the new versions' server.adapter field is null when server.bContinue is false, but the two versions match on the server.adapter field otherwise. It then infers the following conditional transformer, which has the effect of freeing the leaked objects:. server.adapter = null; else server.adapter = server old .adapter;. Note that to generate this transformation function requires using field paths instead of single fields (cf. Section 4.5).. Azureus update SVN r120 Figure 12 shows update r120 to Azureus. This update modifies the condition under which Azureus releases the read buffer between a client and its peer. There are two issues with using TOS to infer this update. The first is that the read buffer does not contain a natural key field. Each buffer is identical and does not have a pointer back to the object using it. This problem manifests as a failure in the matching process. In such a case, one might consider adding a ghost field that records the allocation order and using this as a key field to match objects. Matching succeeds with this addition. However, the assignment of ... } Figure 12 . Azureus update r120. buffers to clients varies across runs (and is unrelated to allocation order), which causes synthesis to fail, since the i th buffer is not associated with the same client in each run. This transformation is thus not inferable from the objects themselves, which is a limitation of our approach. However, it is not surprising that, on occasion, sufficient information for transformation is not available from the heap.. jEdit JEdit is a text editor for programmers that provides common and advanced features, such as syntax highlighting, folding, automatic indentation, a built-in macro language, macro recording, and plugin support. Here we consider two similar memory leaks fixed in jEdit versions r5178 and r14027. Figure 13 shows the source patch for the leak fixed in r14027. jEdit calls the function markTokens when it needs to split a string into tokens based on the type of the file being edited. Each file type (C, Java, Verilog, etc.) has special logic to split text in a line and embed it in an object of type TokenHandler. The leaky jEdit version fails to set the field TokenMarker.tokenHandler to null. The inferred object transformer is simple. It sets the leaky field to null. It is safe to execute the transformer as long as the markTokens function is not active on stack. Figure 14 shows a change to jEdit in SVN revision r13413. The leak is in function HistoryText.showPopupMenu(). jEdit calls showPopupMenu() when the user performs certain actions, such as right clicking a text field with history. In the old version, some instances of HistoryText have the boolean field popup.visible set to true and others set it to false. In the new version, the field popup is not null only when an object also has its popup.visible field set to true. TOS can infer this property and generate a transformer that nulls the popup field of instances that have popup.visible set to false. However, while creating snapshots we were unable to create a situation where popup.isVisible () was true during a snapshot, which would execute instructions controlled by the condition and exercise the leak. This update is an example that TOS is capable of handling, but for which we fail due to test coverage problems. This paper contributes novel matching and synthesis algorithms. The matching algorithm analyzes unstructured heap snapshots from different program version executions. While some recent prior work analyzes a single heap to discover leaked objects and other inefficiencies [2, 3, 9, 11, 12, 15] , none aligns heap objects from different program versions or considers how to fix the effects of leaks on the fly.. A lot of related work considers synthesizing code from specifications, but only recently have researchers considered the problem of synthesizing data transformation functions. The closest related work is by Gulwani and others on synthesizing string and Excel spreadsheet data transformations. These approaches require users to directly specify the input/output examples whereas TOS requires users to run the same test on both program versions from which examples are inferred by matching. Gulwani's algorithm [5] synthesizes string functions that include concatenation, subsequence, and finding special symbols. TOS uses this algorithm as a subroutine as part of synthesizing transformations between objects (cf. Section 4.3). Harris and Gulwani [6] generate transformations between spreadsheets; their numeric transformations and filters are similar to ours. They also search structured spreadsheet data to find correlation between the input and output rows and columns. Our matching phase serves a similar purpose, but once objects are paired up, synthesis follows the structure of the new-version object, assigning its fields one at a time. A unique feature of TOS is that it iterates synthesis and matching to produce transformers for collections of objects.. Many prior dynamic updating systems, including Ginseng [13] , DLpop [8] , POLUS [4] , and Jvolve [14] , provide primitive support for generating state transformation code. For changes that extend classes or structs with new fields, these systems simply copy the old fields and initialize the new ones with default values, e.g., null for object references, or 0 for ints. Systems that provide no direct support for state transformation, e.g., LiveRebel [16] , effectively take this approach. In all of these cases, synthesis is based entirely on comparing the definitions of changed types/classes. None of them consider program semantics by analyzing the code or the heap. By contrast, TOS obtains semantic information from program execution to derive data transformations. In short, while prior systems remove some of the tedium of writing transformation functions, they fail to handle any interesting program changes, which are exactly the cases which are harder for programmers to write correctly. We use the Oracle HotSpot JVM to collect heap snapshots using the agentlib:hprof command line option. 2 This option invokes the heap profiler, which sends the current snapshot over a network socket. We wrote a small server that coordinates snapshots with the application. It initiates snapshots at update points and saves them to disk. In particular, when the application reaches an update point, it calls into a helper class to trigger a snapshot.. TOS is implemented in Java and comprises roughly 4200 lines of code. About 1300 lines implement matching, 1600 implement synthesis, and the rest is common code.

O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Morphogenesis as an amorphous computation In this paper, we present a programming language viewpoint for morphogenesis, the process of shape formation during embryological development. Specifically, we model morphogenesis as a self-organizing, self-repairing amorphous computation and describe a framework through which we can program large-scale shape formation by giving local instructions to cell-like objects. Then, using this programmatic perspective, we specify some example developmental processes and discuss the characteristics that make them suitable candidates for evolutionary variation and selection. Consistent with the theory of facilitated variation from evolutionary biology, we find that variation in developmental processes can be introduced and conserved due to the hierarchical organization of growth specification. Embryological development is a magnificent demonstration of how complexity can arise from initial simplicity. A single egg cell contains most of the information needed to position the millions of cells in a human body; moreover, the construction process is remarkably robust in that it can recover from a large number of cell deaths and malfunctions. Even after the initial construction is completed, the living system remains in a dynamic equilibrium, constantly replacing dead cells, healing tiny ruptures, and so on. Moreover, in some organisms, even large-scale regeneration occurs. Urodeles, like newts, are famous for regrowing entire limbs in such a way that the regenerated parts merge seamlessly into the preexisting tissue.. From the perspective of a computer scientist, studying such processes could bring novel insights into solving other problems, such as programming "smart matter" or directing a swarm of robots. But, even leaving aside these benefits, studying morphogenesis from a computational perspective is important for its own sake: expressing ideas computationally through a programming language forces disambiguation of ideas and provides a powerful tool for examining implications of new hypotheses. A good language in which to express morphogenetic processes will be an invaluable tool to verify plausibility of formal models and to discard inconsistent conjectures. Thus, in the following, the goal is to identify an effective computational framework and a set of morphogenetic primitives through which we can simulate interesting developmental processes.. In section 2, we describe a framework that enables programming of global topological changes through local celllevel instructions. We enumerate a set of basic developmental patterns that performs the role of primitive operations in our language for morphogenesis. Then, in section 3 we present three example developmental mechanisms, reminiscent of ones occurring in nature, and show how they can be implemented in our framework. Finally, we discuss the fact that each of these mechanisms possesses a lot of potential for variation and adaptability and how we can unleash these variations by making small changes to our programs.. cate and kill themselves. The goal of the amorphous system is to attain some desired global state. So, while the organizers are only locally interacting, their collective behavior results in complex global behavior. Moreover, we want the amorphous computation to be robust; that is, the computation should still proceed in the face of random organizer death/failure.. Here, we apply the above computational paradigm to the creation of shapes from amorphous elements. Our goal, as stated above in the Introduction, is to be able to model the essential elements of biological developmental processes with amorphous programs. The amorphous scheme should have the following properties:. • an initially small group of organizers should replicate repeatedly until a desired shape is attained. • organizers should be allowed to move during the course of development. • the shape development should be robust in the face of organizer death and malfunction. However, notice that a trivial development scheme that contains three gradient fields to provide a global coordinate system in a three-dimensional world and that deterministically instructs each organizer to grow another organizer in a particular direction based on the coordinate system can satisfy our criteria. Clearly, there is some other aspect of biological development that is not captured by the above criteria. This, we discuss next.

B O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


The IELR(1) algorithm for generating minimal LR(1) parser tables for non-LR(1) grammars with conflict resolution There has been a recent effort in the literature to reconsider grammar-dependent software development from an engineering point of view. As part of that effort, we examine a deficiency in the state of the art of practical LR parser table generation. Specifically, LALR sometimes generates parser tables that do not accept the full language that the grammar developer expects, but canonical LR is too inefficient to be practical particularly during grammar development. In response, many researchers have attempted to develop minimal LR parser table generation algorithms. In this paper, we demonstrate that a well known algorithm described by David Pager and implemented in Menhir, the most robust minimal LR(1) implementation we have discovered, does not always achieve the full power of canonical LR(1) when the given grammar is non-LR(1) coupled with a specification for resolving conflicts. We also detail an original minimal LR(1) algorithm, IELR(1) (Inadequacy Elimination LR(1)), which we have implemented as an extension of GNU Bison and which does not exhibit this deficiency. Using our implementation, we demonstrate the relevance of this deficiency for several real-world parser specifications, and we demonstrate the feasibility of IELR(1). Finally, we demonstrate that, if canonical LR(1) were employed instead, grammar development would be severely impeded regardless of the power of the computer hardware. Grammar-dependent software is omnipresent in software development [20] . For example, compilers, document processors, browsers, import/export tools, and generative programming tools are used in software development in all phases. These phases include comprehension, analysis, maintenance, reverse-engineering, code manipulation, and visualization of the application program under study. However, construction of these tools relies on the correct recognition of the language constructs specified by the grammar.. Some aspects of grammar engineering are reasonably well understood. For example, the study of grammars as definitions of formal languages, including the study of LL, LR, LALR, and SLR algorithms and the Chomsky hierarchy, form an essential part of most computer science curricula. Nevertheless, parsing as a disciplined study must be reconsidered from an engineering point of view [20, 23] . Many parser developers eschew the use of parser generators because it is too difficult to customize the generated parser or because the generated parser requires considerable modification to incorporate sufficient power to handle modern grammars such as the C++ and C# grammars. Thus, industrial strength parser development requires considerable effort, and many approaches to parser generation are ad hoc [33, 34] .. One source of difficulty in parser development stems from a deficiency in the state of the art of practical LR parser table generation. LR parsing is the most general deterministic shift-reduce parsing method known, and canonical LR is the most general technique for generating LR parser tables from a given grammar [13] . As a result, when a grammar developer considers the general behavior of deterministic shift-reduce parsing without the complexity of any further restriction, the language he then expects his grammar to define is accepted in full by the grammar's canonical LR parser tables. In previous decades, canonical LR parser tables required ''too much space and time to be useful in practice'' [13] , but the validity of this statement is fading with the increasing memory capacity and processing speed of modern computer hardware. However, as our results in Section 4.2 demonstrate, the size of canonical LR parser tables also severely impedes the debugging process during grammar development regardless of the power of the computer hardware.. In contrast, LALR parser tables are practical because they merge canonical LR parser states. Thus, they are employed by widely used tools like Yacc [11, 19] and its GNU implementation Bison [1] . Unfortunately, as Bison's manual points out, this parser state merging causes LALR parser tables to contain ''mysterious conflicts'' that ''don't look warranted'' [17] . These conflicts cause the parser to encounter ''unnatural errors'' because LALR ''is not powerful enough to match the intuition of the grammar writer'' [30, 31] . The reason LALR behavior seems unintuitive to a grammar developer is because, in order to understand the language that an LALR parser accepts for his grammar, he must consider not only the behavior of deterministic shift-reduce parsing but also the complex details of LALR parser state construction and merging. In this way, LALR can worsen the difficulty of developing a correct grammar. Even for an existing correct LALR grammar, LALR can interfere with incremental changes [30, 31] , which are inevitable in the face of software evolution.. In response, many researchers have developed minimal LR 1 algorithms, which attempt to generate parser tables with the power of canonical LR but with nearly the efficiency of LALR [6, 8, [25] [26] [27] [28] 30, 31] . Menhir [7] is an implementation of an algorithm described by David Pager [26] . Initially released in 2005 [12] , Menhir is the most robust minimal LR(1) implementation we have discovered available.. In this paper, we show that Pager's algorithm and Menhir are not always able to generate parser tables with the full power of canonical LR (1) if the given grammar is non-LR(1) coupled with a specification for resolving conflicts. We also describe an original minimal LR(1) algorithm, IELR(1), which does not suffer from this deficiency. In Section 2, we establish a formal theoretical foundation for our IELR(1) algorithm, and we present several non-LR(1) grammar examples that demonstrate the deficiency of Pager's algorithm. In Section 3, we employ the formal models introduced in Section 2 in order to detail our IELR(1) algorithm, which we have implemented as an extension of Bison. We also describe how to modify IELR (1) to generate canonical LR(1) parser tables. In Section 4, we compare the Bison LALR(1) implementation with our IELR (1) and canonical LR(1) implementations using our example grammars plus five parser specifications for popular languages as case studies. In Sections 5 and 6, we review related work and discuss future work. In Section 7, we use the results of our case studies to conclude that (1) Menhir's deficiency does affect real-world parsers, (2) it can create bugs relative to the intended design of such parsers, (3) IELR(1) is feasible for generating minimal LR(1) parsers for sophisticated real-world LR(1) parser specifications, and (4) if canonical LR(1) were employed instead of IELR(1), grammar development would be severely impeded regardless of the power of the computer hardware. Pager's algorithm attempts to avoid LR(1)-relative inadequate states by refusing to merge isocores if its compatibility tests predict that the merge would induce a mysterious conflict somewhere in the parser tables. We use the term predict because the inadequacy may not manifest as a mysterious conflict until a successor state that the algorithm computes later. As we demonstrated in Section 2.6, Pager's compatibility tests fail to predict some mysterious conflicts for non-LR(1) grammars.. In contrast, our IELR(1) algorithm does not try to make predictions in order to avoid inadequate states. Instead, its first phase computes LALR(1) parser tables, which fully merge every isocore set and thus contain some form of every possible grammar-relative inadequacy that can exist after any possible combination of isocore merges. With Observation 2.27 in mind, IELR(1) then computes which of those grammar-relative inadequacies are also LR(1)-relative inadequacies, and it eliminates those that are by splitting inadequate states.. The IELR(1) algorithm consists of 6 phases, which we label phase 0 through phase 5:. • Phase 0: LALR(1). This phase computes the LALR(1) parser tables for G. To do so, it first computes LR(0) parser tables, and it then employs the algorithm of DeRemer and Pennello [16] to compute reduction lookahead sets from goto follow sets, which are sets of tokens that can follow goto actions during a syntactically correct parse. For example, Table 4 shows the cores and actions of interesting LALR(1) states for the grammar in Fig. 5 . It also shows goto follow sets and diagrams their dependencies, some of which must be recomputed in later phases while splitting states. While neither DeRemer and Table 4 . Differences between the parser tables are shown in bold. Given the grammar of Fig. 1 , given that a is left-associative, and assuming an LR(1) parser, the language so specified is a set of 3 sentences: aaa, bab, and baab. As expected, the canonical LR(1) parser tables for this specification accept exactly that language. However, LALR(1) diminishes the language to only 2 sentences: aaa and bab. Thus, by Definition 2.21, LALR (1) is not able to fulfill this LR(1) parser specification. The culprit is not what the Bison manual calls a mysterious conflict, which we call a mysterious new conflict. The culprit belongs to a new category of mysterious conflicts that we call mysterious invasive conflicts. That is, switching from canonical LR(1) to LALR(1) can cause a parser to encounter existing conflicts after viable prefixes for which it never would have encountered those conflicts before and, as a result, to perform incorrect actions. This switch can also cause a parser to perform incorrect actions after viable prefixes for which it would have encountered an existing conflict before. We call this is a mysterious mutated conflict. There are many facets of IELR(1) remaining to be explored. Most importantly, while the results in this paper suggest that IELR(1) can correctbugs in parsers whose specifications are already well evolved with LALR(1), we are especially interested in how IELR(1) facilitates the development of new parser specifications and new formal languages. Thus, we have contributed our IELR(1) implementation to the Bison project and plan to collect feedback from the Bison community. IELR(1) is currently scheduled to appear in Bison 2.5.. In Section 3, we discussed a number of efficiency concepts for IELR(1): split-stable dominant contributions, mergestable dominant contribution functions, and suboptimum state merging. Results gathered from further exploration of these concepts might help us to improve our IELR(1) implementation and might aid developers of alternative implementations of IELR(1).. We are also interested in generalizing IELR(1) to IELR(k) for k ≥ 1. Because the IELR(1) algorithm employs LALR(1), IELR(k) should employ LALR(k). We also suspect that the modifications we described in Section 3.9 for generating canonical LR(1) parser tables via IELR(1) are also appropriate for generating canonical LR(k) parser tables via IELR(k).. Because of the current popularity of GLR (Generalized LR) parsing [32, 35] , it would be useful to explore the benefit of coupling IELR(1) with GLR. A GLR parser can employ parser tables generated by any algorithm in the LR family. Upon encountering an unresolved conflict in those parser tables during a parse, the GLR parser branches and explores all parses to which the conflicting parser actions lead. Thus, for any context-free grammar, if no conflicts in the parser tables are resolved, then all parser actions remain viable, and the GLR parser is able to construct any parse tree specified by the grammar. Even if the parser tables are generated by LALR(1), there is no chance of mysterious behavior because the isocores in any merge are never forced to select potentially different dominant contributions for any conflict. Instead, the isocores always agree that all conflicting actions should be performed.. If a grammar is ambiguous, the GLR parser constructs multiple parse trees for some input sentences. In some cases, it is possible to eliminate some of these parse trees statically simply by resolving the associated conflicts in favour of the desired parse trees. Unfortunately, during LALR(1) parser table generation, the isocores in any merge can then disagree on which contribution to a conflict is dominant. Upon encountering such a resolved conflict, the GLR parser does not branch but behaves in exactly the same mysterious manner as the deterministic LALR(1) parser. In general, when any conflicts in the parser tables used by GLR need to be resolved statically, we conclude that those parser tables should be generated by IELR (1) . One potential improvement to the IELR(1) algorithm presented in this paper would be to ignore unresolved conflicts when annotating states in phase 2 because, again, unresolved conflicts cannot cause mysterious behavior in a GLR parser. We are surprised to discover that mature parser specifications from widely used software products employing LALR(1) parser generators should suffer from any incorrect parser actions that result from the misuse of the LALR(1) algorithm. The Gawk and Gpic case studies provide strong evidence that such incorrect actions do occur in real-world parsers. Such actions are unintuitive and thus may impede the development of a correct parser. Moreover, the Gpic case study shows that such incorrect actions can create actual bugs relative to the intended design of a real-world LALR(1)-generated parser. Pager's algorithm and Menhir do not address the incorrect actions in either of these case studies. IELR(1) corrects them for both.. Because of the maturity of Gpic, we are not surprised that the only bug IELR(1) corrects for Gpic belongs to an obscure feature. Otherwise, we would have expected this bug to have been discovered by now and resolved by other means, such as grammar restructuring or an ad hoc solution. However, the obscurity of this feature does call for further investigation into the general nature of the bugs that IELR(1) corrects for practical parsers.. Because of the maturity of the GCC project, we are not surprised that its C and Java parser specifications suffer from no incorrect parser actions that result from a misuse of the LALR(1) algorithm. Nevertheless, these case studies demonstrate IELR(1)'s ability to recognize when LALR(1) parser tables are sufficient without unnecessarily splitting its states into additional states.. The importance of IELR(1) for our Gawk and Gpic case studies versus our C and Java case studies also suggests that IELR(1) might typically be more relevant to the developer of a DSL (domain-specific language) parser specification rather than to the developer of a GPL (general-purpose programming language) parser specification. This conclusion seems plausible given that the developer of a DSL parser specification is often a domain expert but not a language development expert [24] . By employing IELR(1) instead of LALR(1), Pager's algorithm, or Menhir, a DSL parser specification developer need not struggle with grammar restructuring, ad hoc solutions, and subsequent maintenance in order to manually eliminate mysterious conflicts.. The performance measurements from our case studies show that IELR(1) is feasible for generating minimal LR(1) parsers for sophisticated real-world LR(1) parser specifications. Specifically, for each of our case studies when using the IELR(1) algorithm, Bison did not run longer than 0.3 s, and the IELR(1) algorithm never required as much as 1 MB of memory at any point in time.. Finally, our results demonstrate that canonical LR(1) would severely impede the development of an LR(1) parser specification regardless of the power of the computer hardware. For the practical grammars in our case studies, IELR(1) simplifies the process of debugging parser table conflicts by merging compatible canonical LR(1) states and thus reducing the number of conflicts by an order of magnitude without altering the accepted language. The need to improve the state of the art in grammar-dependent software engineering has been discussed recently in the literature [20, 23, 33, 34] . Many attempts to develop minimal LR algorithms have been made [6, 8, [25] [26] [27] [28] 30, 31] . Menhir [7] is an implementation of an algorithm described by David Pager [26] . It is the most robust minimal LR(1) implementation we have discovered available, but it is not always able to generate parser tables with the full power of canonical LR(1) if the given grammar is non-LR(1) coupled with a specification for resolving conflicts. We previously introduced IELR(1) as a new minimal LR(1) algorithm that does not suffer from this deficiency [15] , but we have not previously defined the algorithm in detail. In this section, we outline phase 3's algorithm in pseudo-code using the functions, tables, and initializations we described in the preceding sections. Phase 3's entry point is the routine split_states, defined in Definition 3.45. Conceptually, the split_states routine propagates lookaheads to successor states recursively starting from the start state. However, the pseudo-code on lines 5-9 split_states propagates lookaheads along these transitions later, and so merge_lookaheads need not recurse any deeper.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B I O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Using templates to distinguish multiplications from squaring operations Abstract Since side channel analysis was introduced as a method to recover secret information from an otherwise secure cryptosystem, many countermeasures have been proposed to prevent leakage from secure devices. Among these countermeasures is side channel atomicity that makes operations indistinguishable using side channel analysis. In this paper, we present practical results of an attack on RSA signature generation, protected in this manner, based on the expected difference in Hamming weight between the result of a multiplication and a squaring operation. This work presents the first attack that we are aware of where template analysis can be used without requiring an open device to characterize an implementation of a given cryptographic algorithm. Moreover, an attacker does not need to know the plaintexts being operated on and, therefore, blinding and padding countermeasures applied to the plaintext do not hinder the attack in anyway. Side channel analysis of cryptographic devices can potentially allow the recovery of secret information by an attacker, even when the underlying algorithms are mathematically secure. An attacker would seek to identify a suitable side channel, such as timing [21] , power [22] , and electromagnetic [14] leakage. Many attacks using these side channels have been described in the literature such as differential [7, 22] and template [9] attacks, as well as countermeasures [10, 12, 20] intended to thwart them.. In [2] , a method of distinguishing between multiplication and squaring operations using acquired traces of power consumption or electromagnetic emanations was demonstrated. In this paper, we extend that work by using template attacks to distinguish between these operations to propose a practical attack. The attack functions with no knowledge of the input to the algorithm, other than that it is random and uniformly distributed. Therefore, the attack is unaffected by padding and/or blinding applied to the input message.. Typically, generating templates to attack a device requires an open device where cryptographic keys are known or can be controlled. The device is then characterized and templates are generated. These templates are then used to derive a cryptographic key on an identical device with very few traces. In this paper, we present an attack where a device with a known key can be used to generate templates. Where a verification function is available that uses the same subfunctions as a signature generation function, this function could be used to generate templates under the assumption that the public key is known.. Previous work applying templates to public key primitives, such as that described in [18, 24] , has targeted the precise Hamming weight of intermediate values. The work described in this paper targets the distribution of the Hamming weight of intermediate values but differs in that the value itself is unknown even when generating the templates. The templates are generated so as to be able to distinguish whether an operation is a squaring operation or multiplication, rather than directly recovering an intermediate value.. The paper is organized as follows: in Sect. 2, the side channel atomic RSA algorithm is described as the target of this work. Further implementation details are given with a description of the Montgomery multiplication algorithm. An overview of template attacks is also given in this section. In Sect. 2.4, the previous work dealing with the the Hamming weight difference between multiplications and squaring operations is revisited. Both simulated and practical results of our attack using templates are provided in Sect. 3. In Sect. 4 , we discuss how the attack would change given a device that leaks information corresponding to the Hamming distance model and demonstrate that the power consumption of the ARM7 used in Sect. 3.2 shows an example of both models. The potential application of our attack to implementations secured with various other countermeasures is described in Sect. 5 . Finally, we draw our conclusions in Sect. 6. In this paper, we demonstrate that the principle of side channel atomicity is not valid simply because the same code is executed for a given function for all possible inputs. This was done by characterizing the difference in expected Hamming weight of the result of a multiplication and squaring operation given random uniformly distributed inputs to generate templates, based on previous descriptions of this difference [2] . These templates are generated by considering single-precision multiplications rather than multiplications between multi-precision values.. These templates can be used to characterize multiplications and squaring operations to determine a private exponent when an RSA signature is generated using Algorithm 1. Previous work in this area has concentrated on building templates on intermediate values (e.g., such as observing where the input value is reused in a left-to-right exponentiation algorithm) rather than the expected distribution of the result of a given function [18, 24] .. The advantage of this work over previous work based on the model originally proposed by Chari et al. [9] is that an attacker does not need an open identical device to conduct the attack. That is, an attacker does not need a device where all the input and cryptographic keys can be changed to arbitrary values. An attacker can use an identical device with a known key, or a verification functions that uses the same operations. Furthermore, the values being operated on do not need to be known, an attacker just needs to know that the values are random.. The proposed attack can be used to derive the private key used in RSA, where one needs to obtain the least significant quarter of an exponent to be able to derive the entire exponent [6] . This naturally extends to the computation of a RSA signature using the Chinese remainder theorem (CRT). Again, one needs to obtain a quarter of the least significant bits of the private key modulo one of the two prime factors of the modulus N (i.e., one of the two exponents used in this algorithm) [5] . The proposed method can also be used to derive a blinded exponent with a certain probability; however, there are currently no results detailing how many bits one would need to identify (i.e., how efficient an attack would be).. The work presented in this paper also applies to elliptic curve scalar multiplication over large prime fields. If strongly unified formulae (e.g., these have been defined for Edward's curves [4] and Weierstraß curves [8] ) are used then an algorithm equivalent to Algorithm 1 can be defined. An attacker could potentially distinguish an addition from a doubling operation using the attack described above. While the bit length of the prime field using in elliptic curve cryptography is significantly smaller than the bit length of an RSA modulus, there are numerous operations in the prime field for each operation, i.e., the addition and doubling operation, that could be exploited.. As noted in [24] , this type of attack naturally extends to attacking implementations of the digital signature algorithm (DSA), and its elliptic curve equivalent (ECDSA) [38] .. In each case, a known value is raised to the power of a random value as part of the signature generation. If some bits of the random value can be determined in one, or several, instantiations of the signature scheme the private key can be derived [27, 28] . The work described in this paper should allow an attacker to derive sufficient bits of this random value to break a naïve implementation of these algorithms with a reasonable probability.. The template attack described in this paper will, counterintuitively, become more effective with longer key lengths due to the extra single-precision multiplications required, as previously noted in [35] for similar reasons. This gives a strong argument for using regular exponentiation algorithms (see [19] for a summary of this topic) rather than side channel atomic algorithms.

O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O B O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Computing median and antimedian sets in median graphs The median (antimedian) set of a profile π = (u 1 , . . . , u k ) of vertices of a graph G is the set of vertices x that minimize (maximize) the remoteness i d (x, u i ). Two algorithms for median graphs G of complexity O(n idim(G)) are designed, where n is the order and idim(G) the isometric dimension of G. The first algorithm computes median sets of profiles and will be in practice often faster than the other algorithm which in addition computes antimedian sets and remoteness functions and works in all partial cubes. The problems of locating medians and antimedians of profiles are natural generalizations of facility location problem in graphs. Given a profile (that is, a multiset of vertices) on a graph/network one wants to locate vertices whose remoteness, that is, the sum of distances to the vertices of the profile, is minimum or maximum. This is a model for those practical problems where vertices of the profile present customers and the resulting median (resp. antimedian) set consists of optimal locations of desirable (resp. undesirable) facilities in the network.. The median problem for profiles on graphs was considered by many authors; see, for example [1, 3, 4, 6, 18] . On the other hand, not much work has been done so far for the general antimedian problem for profiles on graphs. To be exact, there are some papers that deal with the special case of this problem-the obnoxious facility location problem on graphs-where profiles always coincide with the vertex set of a graph; see [7, 9, 20, 22, 24] . We think that the problem of antimedian for profiles could be of similar interest for applications, as well as theoretically, being the opposite extremum to the median problem.. Median graphs form a closely investigated and well understood class of graphs; see [16] for a survey, in particular for their role in location theory. Many important and interesting classes of graphs are median, let us just mention trees, hypercubes, square grids, graphs of acyclic cubical complexes, simplex graphs, and Fibonacci and Lucas cubes. Most of these classes have been considered from the algorithmic point of view. Median graphs themselves can be recognized in subquadratic time [12, 14] , in fact, their recognition complexity is essentially the same as the complexity of recognizing triangle-free graphs [15] , see also [10] . On the other hand, the recognition complexity can be reduced to O(m log n) for acyclic cubical complexes [13] and for Fibonacci cubes [21] . Here and later n and m denote the number of vertices and edges of a given graph. For additional fast algorithms on classes of median graphs see [8] .. The main purpose of this paper is to present efficient algorithm(s) for computing median and antimedian sets of profiles on median graphs. As the main tool we use isometric (that is, distance preserving) embeddings of median graphs into hypercubes. In the next section we fix notation, state necessary definitions, and present a simple algorithm for computing median and antimedian sets in general graphs. In the subsequent section we prove that, given a graph G, embedded via a weak contraction into a graph H , the median set of a profile on G can be obtained from the corresponding median set in H . We elaborate this idea in Sect. 4 for the case when a median graph G is isometrically embedded into a hypercube to obtain a fast algorithm for computing median sets in median graphs. (Unfortunately, a similar theorem does not hold for antimedians.) We follow with another algorithm in Sect. 5 that in addition computes antimedian sets and the remoteness functions within the same time and works in arbitrary isometric subgraphs of hypercubes. In practice, however, the first algorithm will often be faster. In the final section we consider possible generalizations and variations of our approach. Median graphs have many natural generalizations; one such well studied class is the family of quasi-median graphs; see [5] . Among many similarities with median graphs, quasi-median graphs admit isometric embeddings into Hamming graphs. Moreover, they are weak retracts of Hamming graphs [23] , hence Theorem 3.1 can be applied for quasi-median graphs as well. By using a natural extension of majority rule to Hamming graphs, one can easily generalize Algorithm 1 to work for quasi-median graphs.. Step 1 of this algorithm indeed directly extends from hypercubes to Hamming graphs. Having in mind that the vertices of a Hamming graph are t-tuples (whose coordinates are taken from {0, 1, . . . , k i }, respectively), the majority/minority rule is generalized to the set of integers that are most/least frequent in a given coordinate. This implies that in a Hamming graph an (anti)median set induces a Hamming subgraph, and it yields an efficient algorithm for finding (anti)median sets in Hamming graphs. We note that Algorithm 2 also extends to nonbipartite case, because the Hamming distance can be applied in partial Hamming graphs (we leave details to the reader).. In this paper we have embedded median graphs into hypercubes to obtain fast algorithms for computing median and antimedian sets. Since Theorem 3.1 is more general, it can perhaps be applied to other situations as well. In particular it could be useful to embed isometrically median graphs into other Cartesian products, say products of paths or trees. Moreover, one could embed isometrically other classes of graphs into appropriate host graphs, say 1 -graphs.

O B O B O O B O O B B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O B O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O B O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O B I O O O O O O B O O O B O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O


Towards unrestricted lip reading Lip reading provides useful information in speech perception and language understanding, especially when the auditory speech is degraded. However, many current automatic lip reading systems impose some restrictions on users. In this paper, we present our research e orts, in the Interactive System Laboratory, towards unrestricted lip reading. We rst introduce a top-down approach to automatically track and extract lip regions. This technique makes it possible to acquire visual information in real-time without limiting user's freedom of movement. We then discuss normalization algorithms to preprocess images for di erent lightning conditions global illumination and side illumination. We also compare di erent visual preprocessing methods such as raw image, Linear Discriminant Analysis LDA, and Principle Component Analysis PCA. We demonstrate the feasibility of the proposed methods by development of a modular system for exible human-computer interaction via both visual and acoustic speech. The system is based on an extension of an existing state-of-the-art speech recognition system, a modular Multiple State-Time Delayed Neural Network MS-TDNN system. We have developed adaptive combination methods at several di erent levels of the recognition network. The system can automatically track a speaker and extract his her lip region in real-time. The system has been evaluated under di erent noisy conditions such as white noise, music, and mechanical noise. The experimental results indicate that the system can achieve u p to 55 error reduction using additional visual information. The visual information is complementary to the acoustic information complementary in human speech perception, especially in noisy environments. Humans can determine a confused phoneme using both acoustic and visual information because many of the phonemes, which are close to each other acoustically, might b e v ery di erent from each other visually. The connection between visual and acoustic information in speech perception was shown by McGurk with the so-called McGurk E ect 1 . Visual information from the facial region, such as gestures, expressions, head-position, eyebrows, eyes, ears, mouth, teeth, tongue, cheeks, jaw, neck, and hair, could improve the performance of machine recognition 2 . Much research has been directed to developing systems that combine the acoustic and visual information to improve accuracy of speech recognition. These systems mainly focus on integrating acoustic and visual information from the oral-cavity region of a speaker with acoustic information. Two basic approaches have been used in these systems to combine acoustic and visual information. The rst approach uses a comparator to merge the results obtained independently from acoustic and visual sources. The second approach performs recognition using a vector that contains both acoustic and visual information. Most systems reported better performances using both acoustic and visual information than using only one source of information 3, 4, 5, 6, 7, 8, 9, 10 . Most current systems, however, impose certain constraints on users, such as using a head-mounted camera or pacing re ective markers on a user's lips. It is our goal to remove these constraints. In this paper, we present our research e orts towards unrestricted lip reading. Two major reasons cause low quality of visual data in lip-reading: user movement and environment c hange. We present a top-down approach to automatically track and extract lip regions. We use a real-time face tracker to locate a user while the user moves freely. The lip-nder module locates the lips within the found face and provides the coordinates of the mouth corners to lip speech-recognition subsystem, which extracts the relevant information from the image. This technique makes it possible to acquire visual information in real-time without limiting user's freedom of movement. We then discuss normalization algorithms to preprocess images for di erent lightning conditions global illumination and side illumination by comparing di erent visual preprocessing methods such a s r a w image, LDA, and PCA. We show that an adaptive method can automatically adjust parameters to di erent noise conditions. We demonstrate the feasibility of the proposed methods by development of a modular system for exible human-computer interaction via both visual and acoustic speech. The system is based on an extension of an existing state-of-the-art speech recognition system, a modular MS-TDNN system. We have developed adaptive combination methods at several di erent levels of the recognition network. The system can automatically track a user and extract his her lip region in real-time. The system has been evaluated under di erent noisy conditions such a s white noise, music, and mechanical noise. The experimental results indicate that the system can achieve u p t o 5 5 error reduction using additional visual information. Figure 1 gives an overview on the subsystems and their communication of our Lip-reading system. We use a Canon VC-C1 Camera with integrated pan-tilt unit. This unit is controlled by the Face-Tracker module. The Face-Tracker Figure 1 . NLIPS -system overview module sends the position of the face to the Lip-Finder module, which records the acoustic and visual data parallel and stores the position of the mouth-corners for every frame. Tracking of the face and the lip corners is done in realtime during the recording of the data. After that some visual ne-tuning is done to eliminate di erent illumination conditions from the images. The data is then feeded in a MS-TDNN recognizer 11, 12 . All those submodules are described in more detail in the following sections. We h a ve summarized our e orts towards unrestricted lipreading in this paper. Using a top-down approach, a robust real-time tracking system can extract a user lip region while the user moves freely. The illumination changes can be handled e ectively by the adaptation method. We h a ve demonstrated the proposed methods by the continuously spelling task of German letters. The system can achieve u p to 55 error reduction using additional visual information under noisy conditions. We are currently developing a user independent system for language training applications.

O O B I B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O