User evaluation of the NASA technical report server recommendation service We present the user evaluation of two recommendation server methodologies implemented for the NASA Technical Report Server (NTRS).One methodology for generating recommendations uses log analysis to identify co-retrieval events on full-text documents. For comparison, we used the Vector Space Model (VSM) as the second methodology. We calculated cosine similarities and used the top 10 most similar documents (based on metadata) as "recommendations". We then ran an experiment with NASA Langley Research Center (LaRC) staff members to gather their feedback on which method produced the most "quality" recommendations. We found that in most cases VSM outperformed log analysis of co-retrievals. However, analyzing the data revealed the evaluations might have been structurally biased in favor of the VSM generated recommendations. We explore some possible methods for combining log analysis and VSM generated recommendations and suggest areas of future work. NASA's public, web-based digital libraries (DLs) date back to 1993, when a WWW interface was provided for the Langley Technical Report Server (LTRS) [1] . However, it was not until 1995 that the NASA Technical Report Server (NTRS; http://ntrs.nasa.gov/) was established to provide integrated searching between the various NASA web-based DLs [2] . It offered distributed searching, mostly through the WAIS protocol [3] , of up to 20 different NASA centers, institutes and projects. While NTRS was very successful for both NASA and the public, the distributed searching approach proved fragile. In developed. The design and development of the OAI-PMH NTRS is detailed in [5] . A new feature of NTRS is a recommendation service. Based on user anecdotes, we knew the recommendation service was well received, but we desired a more quantitative evaluation of its performance. We would like to repeat this evaluation, but with a larger user group. Unfortunately, there are tradeoffs to overcome in increasing the user size. It is hard to attract the people that are most qualified to rate the evaluations for the NTRS content. Since the staff members are U. S. Government employees, monetary compensation for participation is not bureaucratically feasible. We could do away with in person evaluation sessions and automate the process with features attached to the web page, but then we would risk encumbering the entire NTRS and turn away potential users. The intrusive nature of evaluation is well described by Bishop in the evaluation of the DLI system at the University of Illinois [24] . We are considering contacting aerospace undergraduate and graduate classes and incorporating NTRS awareness with a recommendation evaluation. This would result in more, albeit less experienced, subjects for evaluation.. One of the problems with the spreading activation approach to generating recommendations is the latency between the item entering the system and gathering enough downloads in order to increase the quality of the recommendations. This problem is two fold: first, logs are collected from NTRS and processed on a monthly basis, causing at least a 1 month delay before an item can be eligible for recommendations; second, items with "low popularity" (i.e., few downloads) can be in the system for many months before the quality of their recommendations starts to stabilize. We are experimenting with approaches to seed the recommendation process with VSM results, and then let spreading activation guide the recommendation process afterwards.. We are experimenting with approaches to minimize the impact of robots on the system. Large-scale robot downloads of the eprints in NTRS can create and reinforce links that are artifacts of accession order and do not represent semantic relationships. Only one robot, a LaRC robot affiliated with another project, was excluded from generating the current recommendations.. While some robots identify themselves through the HTTP_REFERER field, many do not in order to avoid dynamic anti-robot tactics (i.e., "spider traps", HTTP status-code 503 (Service Unavailable), etc.). We will include in our log processing facility the ability to identify and discard repeated, rapid downloads that do not represent interactive user sessions. Another approach to lessen the impact of automated access would be to re-run the test using only the most popular documents (e.g., the first 20 downloads shown in figure 4). We have described NTRS and the architecture of the NTRS recommendation server. We performed a quantitative analysis of the recommendation server and compared against baseline recommendations generated by VSM.. The log analysis recommendation method did not perform as well as the VSM recommendation method, but the log analysis method was handicapped by a number of factors. The test documents were chosen randomly from the LaRC portion of NTRS, and not all documents had received enough downloads to have mature recommendations. Secondly, it was difficult to get a good match between document subject area and user expertise. The expertise of the evaluation subjects was largely outside the core focus area of NTRS. For future evaluations, we will seek out more participants who are better situated to review the subject material, even if they are less experienced. We are investigating methods to lessen the impact of robots and to seed the log analysis recommendations with VSM results. The evaluation sessions resulted a total of 129 observations, i.e. individual comparisons of the quality of recommendations issued by method A and method B for a specific document. In total, 129 comparisons pertained to a set of 29 documents. The results were tabulated so that each row of the resulting data set contained the document identifier, the rater identifier, results for method A, method B and the self-reported level of rater expertise. All ratings were reported on a 10 point scale, 0 corresponding to no qualitatively adequate recommendations, 10 indicating all recommendations generated by a particular method are of high quality. Descriptive statistics were generated over all ratings for method A and B, and are listed in Table 4 . The mean ratings for method A (log analysis) and method B (VSM) were 2.28 and 6.9, respectively, and thus diverged considerably. Method B outperformed method A, but the standard deviations of the distributions of ratings indicate both methods may outperform the other in particular instances.. An ANOVA (analysis of variance) was performed over the distribution of rating values to determine whether their means were significantly different. In this case, the null-hypothesis (the means of the two method ratings are not statistically different) was rejected at the p<0.1 level, indicating the means are marginally different. This result is probably caused by the wide dispersion of the rating values. Figure 3 shows the frequency distribution of the observed ratings for method A and B which indicate that both methods perform at similar levels for a significant number of documents, but that the ratings definitely favor method B over method A.. Since our raters indicated their knowledge level on the document for which recommendations were issued, we determined the degree of relationship between rater knowledge and recommendations ratings. It is conceivable that more knowledgeable raters prefer one method to the other. For each knowledge level ranging from 0 (layman) to 5 (expert) we determined the mean rating for recommendations issued by method A (log analysis) and method B (VSM). The results are listed in Table 5 and indicate that method B is preferred by raters at all knowledge levels, although roughly 2/3s of raters indicate their domain knowledge is on the layman level. Method A is rated best but still below method B for low knowledge levels. Although both methods are rated lower by expert raters, method B has a strong preference among that group as well.. Spearman correlation coefficients were calculated to determine the degree of relationship between rater domain knowledge and method ratings. The correlation between method A ratings and rater domain knowledge was ρ=-0.156 (df=129, α<0.1) indicating there is a weak but marginally statistically significant negative relationship between rater domain knowledge and method A ratings. For method B and rater domain knowledge we found a Spearman's ρ=0.12931 (df=129, α>0.1) indicating the absence of a statistically significant relationship between rater domain knowledge and method B ratings. Method A and method B ratings over all rater domain knowledge levels was found to be ρ=0.20100 (df=129, α<0.0.5) indicating a positive and statistically significant relationship between method A and method B ratings. The latter result indicates that some documents cause both methods to produce recommendations that are favorably rated.. An important matter of concern in the use of log analysis to produce document recommendations is the availability of sufficient usage data and its distribution. Document usage can be expected to follow an inverse power law where some documents are retrieved very often and many others only sporadically. However, figure 4 shows that after about the top 20 downloads, NTRS retrieval patterns do not follow a power law. This could be due to robot access to NTRS and only the most popular documents emerge from the cloud of evenly distributed web crawler accesses. The nature of method A's reliance on user retrieval patterns for the generation of document recommendations will naturally lead it to produce a higher number of more valid recommendations for frequently retrieved documents. Our sample of random documents may thus not accurately reflect the performance of method A under realistic circumstances where users would tend to frequently download a particular set of documents for which method A has the highest number of valid recommendations. It forces method A to compete with method B in the absence of retrieval data leading to an invalid comparison on grounds of missing data.. To test this hypothesis we retrieved the absolute number of recommendations available for each document under method. Due to our log analysis method this number corresponds to frequency of use. We then correlated the number of recommendation for each document with its ratings under method A and method B. Indeed, we found a statistically significant relationship between the ratings of method A and the number of available recommendations (ρ=0.201, df=129, α<0.05). This result indicates that as more recommendations are available method A is rated higher. The rater preference of method B over method A can thus largely be caused by the absence of sufficient log data for the documents used in the evaluation. In addition, we found a negative but statistically significant relation between method B ratings and the number of recommendations available for recommendations A (ρ=-0.32,df=129, α<0.05). Since the latter corresponds to document usage, we must conclude that method B produces less valid recommendations for more often retrieved documents. We have not yet formulated a hypothesis to explain this result, but speculate that often retrieved documents may be of a more general nature, and will therefore lead VSM recommenders to discover fewer valid document relationships due to the lack of precise term-relationships. On the other hand, method A relies on actual usage and will simply adopt whichever document relationships are favored by users in their actual retrievals.. To further explore this concept, we determined the ratio of the ratings of method A and method B, a metric indicating how strongly one method is preferred over the other, and the number of method A recommendations available. As expected, we found a strong and statistically significant relation (ρ=0.384, df=129, α<0.01) indicating that where sufficient log data is available, method A will increasingly improve its recommendations relative to those of B. Digital library evaluation is an area of growing interest. Presumably, this stems from the proliferation of DLs and their creators, managers and funding parties wanting to know "what is happening with our DL?" Jones et al. describe transaction analysis of the New Zealand Digital Library and derive interface suggestions from looking at user sessions and common mistakes [14] . Sfakakis and Kapidakis [15] studied the log files of the Hellenic Documentation Centre Digital Library and found users favor simple queries. Steinerova [16] describes DL usage patterns of Slovakian DL users as reported through questionnaires. Assadi et al. [17] describe DL usage patterns in France through surveys, interviews and instrumented DLs. Saracevic [18] offers a broad conceptual framework for DL evaluation.. There is an equal amount of work on the evaluation of recommendation systems. A number of studies have focused on improving existing recommendation systems, such as Efron & Geisler [19] and Hofmann [20] using singular value decomposition to improve recommendations. Kautz, Selman and Shaw [21] describe exploiting existing social networks to increase the quality of recommendations. Sinha and Swearingen [22] compared automated and human generated recommendations and although human recommendations were preferred, the automated recommendations often provided interesting and novel recommendations. Herlocker et al. [23] provide a comprehensive framework for evaluating recommendation systems. We designed an experiment to evaluate the NTRS recommendation server using the domain knowledge of researchers at NASA Langley Research Center (LaRC). Using the terminology from Herlocker et al.'s framework, we designed a "find good items" user task. Twenty documents from the LaRC portion of NTRS were chosen at random. We verified that recommendations were available for these documents through the log analysis-spreading activation (labeled "log analysis") method. For each of the LaRC documents in NTRS (approximately 4100), we calculated the top 10 most similar documents (according to the Vector Space Model) from the entire NTRS corpus. A program was written on a separate web site to allow the users to selfidentify, and then step the user through each of the 20 documents. For each test document, an abstract was shown (with a link to the full-text document) and links were provided for the top 10 recommendations as computed by "Method A" (log analysis) and "Method B" (VSM). Figure 2 shows a screen shot of the data collection web page and the 2 pop-up windows with recommendations from methods A & B. Method A is the recommendation system in use in the production version of NTRS, but it was not described as such to avoid biasing the results.. The data collected for each document included the user's level of expertise (1..5) with the document's subject area, and the perceived number (0..10) of "quality" recommendations as generated by each method. A text box was included for any freeform comments the users wished to contribute. Before each session, the users were given a short preparatory presentation that included the history of NTRS, the purpose of the evaluation, and several ways to judge the quality of recommendations. We acknowledged to the participants that "quality" is largely subjective, but nonetheless suggested to consider such factors as: A call for participation was emailed to targeted organizations and posted on the LaRC intranet to solicit volunteers for one of the four 90 minute sessions, and a total of 13 volunteers were recruited. The volunteer profile is summarized in Table 1 , and their organizational (and thus subject area expertise) is summarized in Table 2 . The sessions were held at LaRC in a separate computer based training facility away from the participants' normal offices. No monetary compensation was given to the participants, but refreshments were served. The actual test session was divided into two parts: everyone evaluated the same set of twenty documents, and then after completing that part, they could search NTRS for their own (or "favorite") document and then rate the recommendations by both methods for those documents. During the first session it became apparent that the users would not be able to do 20 evaluations in the allotted time, so the test set was truncated from 20 to 10 test documents. Evaluations for 29 documents were generated from the list of 10 standard document and the documents the users found themselves.. The subject areas of the 29 documents are summarized in Table 3 .

O B O O O O O B B O O O O O B O O B B O O O O O O O B O B O O O O B I O O O O O O O O O O O O B I I O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O B O O B O O O B O O O O O O O O O B I O O O O O B O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B O O O O B I O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O B B O O O O O O O O O B I B B O O O O O O O O B B O O B I B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B O O O O O O O O O B O O B O O O O B O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O B O O O O O O O O O O O O O O B O B I O B O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O B I O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O B O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O B O O O O O B B O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O B I B O O O O O O O O O O O O O B O O O O O O O B O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O B O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O B O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Diconic addition of failsafe fault-tolerance We present a divide-and-conquer method, called DiConic, for automatic addition of failsafe fault-tolerance to distributed programs, where a failsafe program guarantees to meet its safety specification even when faults occur. Specifically, instead of adding fault-tolerance to a program as a whole, we separately revise program actions so that the entire program becomes failsafe fault-tolerant. Our DiConic algorithm has the potential to utilize the processing power of a large number of machines working in parallel, thereby enabling automatic addition of failsafe fault-tolerance to distributed programs with a large number of processes. We formulate our DiConic synthesis algorithm in terms of the satisfiability problem and demonstrate our approach for the Byzantine Generals problem and an industrial application. Software intensive systems are inevitably subject to the occurrence of unanticipated faults as it is difficult (if not impossible) to anticipate all types of faults in early stages of development. Automatic addition of fault-tolerance to programs provides a systematic approach for equipping an existing program with necessary fault-tolerance functionalities while preserving its correctness in the absence of faults. However, the exponential complexity of automatic addition * This work was sponsored by a grant from Michigan Technological University.. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. of fault-tolerance to distributed programs [22] urges us to devise new paradigms for the addition of fault-tolerance that scale better than existing techniques. In this paper, we present a DIvide-and-CONquer paradigm, called DiConic, that decomposes the problem of automatic addition of failsafe fault-tolerance -where a failsafe program meets its safety specification even when faults occur -into a set of sub-problems, where in each sub-problem, we focus on revising an individual program action (specified in terms of Dijkstra's guarded commands language [9] ) in such a way that the entire program becomes failsafe fault-tolerant.. Most existing approaches [17, 25, 23, 28, 8, 29, 5, 18, 20, 13, 7] for automated synthesis of fault-tolerant programs generate an integrated model of a program (implemented as a reachability graph in enumerative techniques [18, 20, 13] or as a set of Binary Decision Diagrams (BDDs) in symbolic methods [29, 7] ) which makes it difficult to scale such methods as the size 1 of the program increases. For example, specification-based approaches [17, 5] synthesize the abstract structure of a fault-tolerant program as a finitestate machine from the satisfiability proof of its temporal logic specification. Control-theoretic techniques [23, 8] use synchronous automata-theoretic product to generate finite state automata that represent fault-tolerant discrete-event controllers. Game-theoretic methods [25, 28] synthesize a winning strategy that captures all program behaviors. Previous work on addition of fault-tolerance to concurrent and distributed programs [18, 20, 13] takes an integrated model (represented as a finite-state machine) of an existing program and automatically revises the program model in order to generate a model of a fault-tolerant version thereof. Symbolic techniques [29, 7] enable the synthesis of larger programs, however, they still suffer from the complexity of synthesis as they deal with the synthesis problem as a whole without decomposing the problem. While all aforementioned approaches present important and useful techniques for synthesizing fault-tolerant programs and mitigating space/time complexity of addition, we believe that decomposing the synthesis problem so that each sub-problem can be solved separately is equally important and is orthogonal to complexity issues.. In order to decompose the problem of automatic addition of failsafe fault-tolerance, we focus on the following questions: Given a fault-intolerant program that satisfies its specification in the absence of faults, but provides no guaran-tees in the presence of faults, how can we determine whether or not each program action should be revised so that the entire program becomes fault-tolerant? If indeed a program action should be revised, how do we independently identify the nature of the revision for that action? These questions are indeed special cases of a more general question, which is the focus of our research: Given a program that meets its specification and a new desired property raised due to the change in requirements, how do we revise each program action so that the revised program satisfies the new property while preserving its existing specification? To address this question for the addition of failsafe fault-tolerance, we present a sound distributed algorithm that has two components: synthesis coordinator and synthesis node. Corresponding to each program action, we instantiate a synthesis node that determines how that action should be revised towards the synthesis of a failsafe version of the input program. The synthesis coordinator synchronizes the activities of synthesis nodes. There is only one synchronization point between a synthesis node and the synthesis coordinator, which helps decreasing the communication cost of our algorithm. As the number of program actions increases one can instantiate new synthesis nodes and deploy them on different parallel machines.. Our proposed synthesis algorithm is a distributed fixpoint computation where in each round of the computation synthesis nodes report their results to the synthesis coordinator. More specifically, our algorithm takes a fault-intolerant program (as a set of actions) and a set of fault actions, and instantiates (i) N synthesis nodes, where N is the total number of program and fault actions, and (ii) one coordinator node. Each synthesis node performs a set of verification and revision operations that could be done in parallel with revisions performed on other actions in other synthesis nodes. Afterwards, each synthesis node reports its revised action to the coordinator node and waits for coordinator's reply. At the end of each round, the collection of the actions in all synthesis nodes comprises an intermediate program. The coordinator verifies some constraints on the intermediate program and broadcasts the results to all synthesis nodes that are waiting to enter the next round of synthesis. A synthesis node terminates in two possible cases. First, the synthesis node locally determines that its associated action should be removed in the failsafe program. Second, the synthesis node receives a termination message from the coordinator. The coordinator terminates either by finding a failsafe program or by declaring failure.. We have used the Yices Satisfiability Modulo Theories (SMT) solver [10] (developed at SRI International) and symbolic techniques to apply our DiConic approach to several case studies (available in [12] ) among which (i) the classic problem of Byzantine Generals (BG) [24] , and (ii) a simplified version of an altitude switch controller program that is similar to its manually designed version (developed at the Naval Research Laboratory [6] ). The organization of the paper is as follows: We present preliminary concepts in Section 2. In Section 3, we represent the problem of adding failsafe fault-tolerance. Subsequently, in Section 4, we reiterate an enumerative solution due to Kulkarni et al. [19] where they add failsafe fault-tolerance to distributed programs represented as finite-state machines. Then, in Section 5, we present our DiConic algorithm. We use the BG problem as a running example to demonstrate our approach. In Section 6, we use our DiConic approach to add failsafe fault-tolerance to a simplified version of an altitude switch controller (from [6] ). We summarize our case studies in Section 7. We discuss related work in Section 8. Finally, we make concluding remarks and discuss future work in Section 9. We presented a divide-and-conquer approach, called DiConic, for the addition of failsafe fault-tolerance to (distributed) programs, where in the presence of faults a failsafe program guarantees to satisfy at least its safety specification. We specifically focused on the following question: Given a fault-intolerant program and a specific fault-type, how can we revise each program action separately so that the entire program becomes failsafe fault-tolerant? To address this question, we decomposed the problem of adding failsafe faulttolerance into sub-problems. In each sub-problem, we concentrated on one program action and determined how that action should be revised so that the entire program would be failsafe fault-tolerant. We validated our approach for the classic problem of Byzantine Generals and for a simplified version of an altitude switch controller.. As an extension of this work, we are investigating the DiConic addition of recovery to programs, where we revise program instructions in isolation so that the entire program would eventually recover to its invariant after faults stop occurring. Another extension of this work is to enhance the efficiency of the synthesis coordinator by developing its distributed version. DiConic addition of fault-tolerance is a special case of a more general problem in which new Temporal Logic [16] properties are incrementally added to an existing program (as defined and addressed in our previous work [15] ). Towards this end, we plan to develop new DiConic algorithms for automatic addition of new properties to (multithreaded and distributed) programs. In this section, we discuss related work on existing divideand-conquer synthesis methods [27, 26] and techniques for reducing the time/space complexity of automatic addition of fault-tolerance [19, 20, 21, 14, 7] . Specifically, Smith [27] presents a divide-and-conquer approach for synthesizing programs from their specifications, where he decomposes the program specification into the specifications of sub-problems and combines the results of synthesizing solutions for subproblems. Puri and Gu [26] propose a divide-and-conquer synthesis method for asynchronous digital circuits, where they decompose circuit specifications and satisfy design constraints for each sub-specification. While the aforementioned approaches inspire our work, they are essentially specification-based approaches and do not directly focus on adding failsafe fault-tolerance to existing programs.. Previous work on automatic addition of faulttolerance [19, 20, 21, 14, 7] mostly focuses on techniques for reducing time/space complexity of synthesis. For example, Kulkarni et al. [19] present a set of heuristics based on which they reduce the time complexity of adding fault-tolerance to integrated models of distributed programs. Kulkarni and Ebnenasir [20] present a technique for reusing the computations of an existing fault-tolerant program in order to enhance its level of tolerance. They also present a set of pre-synthesized fault-tolerance components [21] that can be reused during the addition of fault-tolerance to different programs. We have presented a SAT-based technique [14] where we employ SAT solvers to solve some verification problems during the addition of fault-tolerance to integrated program models. Bonakdarpour and Kulkarni [7] present a symbolic implementation of the heuristics in [19] where they use BDDs to model distributed programs.. Our DiConic approach significantly differs from the previous work in that we present a new paradigm for decomposing the problem of adding failsafe fault-tolerance instead of adding failsafe fault-tolerance to an integrated model of the entire program. More specifically, time/space complexity issues are orthogonal to our DiConic approach since one can benefit from existing techniques to mitigate the complexity of revising individual program actions.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


A formal theory for optimal and information theoretic syntactic pattern recognition In this paper we present a foundational basis for optimal and information theoretic syntactic pattern recognition. We do this by developing a rigorous model, M * , for channels which permit arbitrarily distributed substitution, deletion and insertion syntactic errors. More explicitly, if A is any finite alphabet and A * the set of words over A, we specify a stochastically consistent scheme by which a string U ∈ A * can be transformed into any Y ∈ A * by means of arbitrarily distributed substitution, deletion and insertion operations. The scheme is shown to be Functionally Complete and stochastically consistent. Apart from the synthesis aspects, we also deal with the analysis of such a model and derive a technique by which Pr[Y|U], the probability of receiving Y given that U was transmitted, can be computed in cubic time using dynamic programming. One of the salient features of this scheme is that it demonstrates how dynamic programming can be applied to evaluate quantities involving complex combinatorial expressions and which also maintain rigid probability consistency constraints. Experimental results which involve dictionaries with strings of lengths between 7 and 14 with an overall average noise of 39.75 % demonstrate the superiority of our system over existing methods. Apart from its straightforward applications in string generation and recognition, we believe that the model also has extensive potential applications in speech, uni-dimensional signal processing, and computational molecular biology. In the field of statistical Pattern Recognition (PR), the patterns are represented using numerical features. As opposed to this, in syntactic and structural PR the classifiers are designed to be trained and tested by representing the patterns symbolically using primitive or elementary symbols. The PR system essentially models the noisy variations of typical samples of the patterns symbolically, and these models are utilized in both the training and testing phases. In statistical PR, the noisy samples from a given class are modeled (either parametrically or non-parametrically) using probability distributions, the class conditional distributions. It is well known [1, 2] that if these distributions are known, information theoretic, minimum probability of error classifiers can be designed.. In this paper we shall attempt to formally lay the foundation for optimal 3 syntactic PR systems which permit arbitrarily distributed noise 4 . To achieve this we have, in a novel way, "decoupled" the occurrence of noisy observations into two distinct modules. The first of these formally describes the module which accounts for the "dictionary" of ungarbled observations and the second describes the channel which syntactically corrupts error-free signals. This approach differs significantly from the original traditional way by which the problem was approached by the pioneer Fu 5 [3] and his co-authors. The latter represented the set of possible garbled occurrences of possible patterns using stochastic string grammars from the wellknown Chomsky hierarchy, or by web or tree grammars.. The alternate strategy for designing syntactic PR recognition systems works as follows. The system has a dictionary 6 which is a collection of all the ideal representations of the objects in question. When a noisy sample has to be processed, the system compares this sample with every element in the dictionary. This comparison is either done sequentially or using a grammatical parsing mechanism. The question of comparing patterns reduces to one of comparing their string representations, and this is typically achieved using three standard edit operations -substitution, insertion and deletion. To achieve this, one usually assigns a distance for the elementary symbol operations, and the inter-pattern distance is computed as a function of these elementary symbol edit distances. Recognition using distance criteria is obtained by essentially evaluating the string in the dictionary which is "closest" to the noisy one as per the metric under consideration.. The elementary distances can be assigned weights in a variety of ways. If R + is the set of nonnegative real numbers, the elementary edit distances are typically defined using three elementary functions d s (.,.), d i (.) and d e (.) :. 3 By "optimal" we mean that we can obtain decisions which minimize the probability of error or which minimize the total expected risk. As we will see later, the conditions required for information theoretic syntactic PR are more stringent. 4 In this paper we shall only deal with syntactic PR of patterns which are represented "linearly" as strings. The problem of developing information theoretic classifiers for PR systems using two-dimensional structures such as trees and webs remains open. Some initial results for the case of ordered tree representations are currently available. 5 Prof. Fu (to whose memory we have dedicated this paper) has authored or co-authored dozens of papers and books in this area. We recommend [3] as an ideal reference for the entire field as it was studied in its infancy. 6 A discussion of the various dictionary models encountered in syntactic PR follows presently. Syntactic Pattern Recognition. Page 3 The parametric string distance has some amazingly interesting properties derived in [4] . The assignment of 'r' and the application of the inter-string distance in PR has also been alluded to in [4] .. If, however, the elementary symbol edit distances are symbol dependent, the distance is called the Generalized Levenshtein Distance 7 . The question of how the elementary symbol edit distances can be assigned is relatively open; indeed, they can be parametrically assigned as in [4] or can be related to the inter-symbol confusion probabilities via their negative logarithms as recommended in [5, 6] . The explicit form of the individual edit distances often takes the form : The fundamental problem that arises from all the above three assignment strategies is that the final classified string obtained using such edit distances has no probabilistic significance except in some rather simple cases. Furthermore, if D(X, Y) is the edit distance associated with editing X to Y, the latter has no explicit relationship to Pr(X → Y) except in a few rather trivial cases.. A little insight into the problem would reveal that the fundamental question which traditional strategies avoid is one of stochastically modeling the structural behaviour of the patterns. Viewed from a reverse engineering (black-box) perspective this question is one of specifying how the individual patterns from the various classes could have been generated, an understanding of which could lead to the designing 7 The terminology used in the literature varies. Throughout this paper, we shall categorically use these terms.. Optimal and Information Theoretic Syntactic Pattern Recognition. Page 4 of optimal classifiers. This is the central problem studied in this paper. Consequently, we shall endeavour to present a new model for noisy channels which transfer (or rather, carry) symbolic data, garbling it with arbitrarily distributed substitution, deletion and insertion errors. To our knowledge, this model, M * , is the first generalized model of its type 8 . Indeed, we also show that if we compare our model with all other channel models which have the same common underlying garbling philosophy -(that transmitted symbols can only be substituted for or deleted, and received symbols are obtained as either a result of transmitted symbols being substituted for or as inserted symbols), our model attains the information theoretic upper bound for the recognition accuracy.. Apart from its straightforward applications in string generation and recognition, we believe that like its predecessors [7,8,9,10 ,11,12,13,14,15,16] , the model also has potential applications in speech, unidimensional signal processing, and in molecular biological comparisons. In this paper we have presented a formal foundation for designing optimal and informatic theoretic syntactic pattern recognizers. We have done this by presenting a new model for noisy channels which permit arbitrarily distributed substitution, deletion and insertion errors. The model is specified in terms of an excited noisy string generation technique. Given any arbitrary string U ∈ A * , we specify a stochastically consistent scheme by which this word can be transformed into any Y ∈ A * by causing substitution, deletion and insertion operations. The scheme has been shown to be Functionally Complete because it involves all the ways by which U can be mutated into Y using these three operations. The probability distributions for these respective operations can be completely arbitrary. Apart from presenting a scheme by which all the possible strings in A * can be potentially generated, we also specify two cubic-time algorithms by which Pr[Y|U], the probability of receiving Y given that U was transmitted, can be computed. The first of these requires cubic space, and the second requires only quadratic space. For small values of M and N, the former is more efficient, because of the decreased overhead and book-keeping. The classification obtained using this model is not only optimal, but it also attains the information theoretic bound for channels which substitute or delete its transmitted symbols, and whose received symbols are either inserted or substituted versions of transmitted symbols.. Since the submission of this paper we have solved the problem of designing optimal and information theoretic syntactic classifiers for traditional and generalized transposition errors. In both these cases the estimation (inference) problem remains unsolved. The question of determining how our current model compares with HMM models in actual applications remains open. Finally, we are currently investigating how the fundamental results presented here can be extended to "continuous time" symbolic computations.

O O O O O O O O B I I O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Toward generating reducible replay logs Logging and replay is important to reproducing software failures and recovering from failures. Replaying a long execution is time consuming, especially when replay is further integrated with runtime techniques that require expensive instrumentation, such as dependence detection. In this paper, we propose a technique to reduce a replay log while retaining its ability to reproduce a failure. While traditional logging records only system calls and signals, our technique leverages the compiler to selectively collect additional information on the fly. Upon a failure, the log can be reduced by analyzing itself. The collection is highly optimized. The additional runtime overhead of our technique, compared to a plain logging tool, is trivial (2.61% average) and the size of additional log is comparable to the original log. Substantial reduction can be costeffectively achieved through a search based algorithm. The reduced log is guaranteed to reproduce the failure. Logging and replay is an important technique for software dependability [6, 11, 20] . It records the interactions between a program and its environment during execution by logging system calls and signals. The execution can be replayed as many times as necessary for various purposes, such as diagnosis of software failures and state recovery from failures. The state of the art logging techniques have low overhead, usually less than 10% [6, 18] , support concurrent programs [1, 5, 16] , and are even provided as part of the operating system. These features make the techniques highly desirable when intensive on-the-fly human attention is infeasible during execution, such as in the emerging cloud computing scenario.. For long running programs such as server programs and user interactive (UI) programs, replay logs may correspond to executions as long as a few hours or even days. Large logs entail long replay times. The programmer may have to wait for hours before a failure is reproduced. Sending such large log files over network could Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. PLDI'11, June 4-8, 2011 , San Jose, California, USA. Copyright c 2011 ACM 978-1-4503-0663-8/11/06. . . $10.00 also be problematic. Checkpoints [4] are often created to mitigate the problem. However, as checkpointing often entails taking snapshots of the entire address space of the application, its frequent use cannot be afforded. We aim to develop a practical technique that reduces an execution by reducing its replay log. Given a reduction criterion, such as a program failure, the technique removes events from the log in a way that the reduced log can still properly drive the execution to the criterion. Such a technique is particularly desirable when replay is integrated with debugging techniques that require expensive instrumentation easily causing an order of magnitude slow down [25] . There are also iterative debugging techniques that require repeated replay [24] . In such a context, even a log for a few minutes of execution is hardly affordable. Furthermore, if a failure occurs in remote execution, the large replay log can be reduced on the remote site before it is sent back to the developer. We propose a compiler based technique that generates a reducible replay log. The technique divides an execution into units, mainly iterations of event processing loops. We instrument programs instrument to collect minimal additional information like memory accesses into the replay log. Given a criterion, reduction can be achieved through analyzing just the log. Our technique is automated, only requiring annotating the event processing loops. It is highly efficient: the average runtime overhead is 2.6% and the additional space consumption is comparable to logs containing only syscalls and signals. Our reduction and replay scheme is also novel. It seamlessly weaves value based replay and dependence based replay to achieve both great reduction and faithful reconstruction of memory state. Our results show that we can reduce executions with up to 2,033K units to less than 91 units. Execution fast forwarding (EFF) [19, 25] also has the goal of reducing logs. EFF does not instrument the program during the logging phase. Instead, it analyzes dependences between event entries offline, either through static analysis or by replaying the whole execution once to detect dependences, which is too expensive (usually 10X slowdown). EFF does not guarantee replayability. In this paper, we show that with small cost, we can acquire a reducible log on the fly. Reduction can be achieved by analyzing the log, without static analysis or replaying the full execution. We also observe that EFF is too conservative in considering all dependences, often leading to very little reduction. In contrast, we weave value and dependence based replay to a novel and highly effective scheme.. Language based replay [21] allows users to replay a program component (module). It uses profiling to find a replay cut to minimize logging efforts. Whereas the technique can be considered a reduction on the program dimension, our reduction is on the temporal dimension, and we believe the two dimensions are orthogonal. Moreover, to use their technique, the user has to know about where a bug might be to do selectively logging. In comparison, our logging is general. We also handle failures that cross components. SCARPE [10] is a similar component based technique for Java. Subgroup replay [23] groups processes and records only intergroup messages.. Checkpointing [4] is a standard approach to avoid replaying a whole execution. Incremental checkpointing [3, 15] avoids overhead from capturing memory snapshots. Language based checkpointing [22] allows users to checkpoint at arbitrary program points with contexts. Our technique is complementary to checkpointing, e.g., our technique can further reduce an execution between checkpoints. Furthermore, we allow fine-grained reduction at the unit (loop iteration) level.. There are software based replay systems that record individual memory accesses and their happens-before relations [5] . Such systems induce substantial runtime overhead. There has been substantial work on software based record and replay for parallel and distributed systems [2, 6, 11, 17, 18] . These systems only perform coarse-grained logging at the level of system calls or control flow and hence are not sufficient for reduction. Hardware based logging and replay [7, 8, 12, 14] can faithfully replay executions. While such techniques are effective, they demand special hardware.. In recent years, significant progress has been made in testing and debugging concurrent programs [1, 9, 13, 16] . These techniques search for a failure inducing schedule given certain inputs. In the future, we plan to leverage these techniques to generate reducible logs on multiple cores.. Delta debugging [24] reduces a failure inducing input by repeatedly running the program on subsets of the input. The process could be expensive for long execution.

O O O B O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O B O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O


Generalized

Analytic

Rule

Extraction

for

feedforward

neural

networks

Abstract-This

paper

suggests

the

"lnpUt-NetwOik-Training-0ulput-Extract~~-Knowledge"

framework

to

classify

existing

rule

exlraction

algorithms

for

feedloward

neural

networks.

Based

on

the

Suggested

framework,

we

identify

the

major

practices

of

existing

algorithms

as

relying

on

the

technique

Of

generate

and

test,

which

leads

to

exponential

complexity,

relying

on

specialized

network

structure

and

training

algorithms,

which

leads

to

limited

applications

and

rsliance

on

the

interpretation

ot

hidden

nodes,

which

leads

to

proliferation

of

classification

ruies

and

their

incomprehensibility.

in

order

to

generalize

the

applicability

of

rule

extraction.

we

propose

the

rule

extraction

algorithm

GeneraLized

Analytic

Rule

Extraction

(GLARE),

and

demonstrate

its

efficaoy

by

comparing

it

with

neural

networks

per

se

and

the

popular

rule

extraction

program

for

decision

trees,

Term-Ciassiticatian,

neural

network,

rule

extraction.

O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B I O O O B I O O O B I O O O O O O O O O O O O O O O O B I O O O O O O B I B I


Effects for cooperable and serializable threads Reasoning about the correctness of multithreaded programs is complicated by the potential for unexpected interference between threads. Previous work on controlling thread interference focused on verifying race freedom and/or atomicity. Unfortunately, race freedom is insufficient to prevent unintended thread interference. The notion of atomic blocks provides more semantic guarantees, but offers limited benefits for non-atomic code and it requires bimodal sequential/multithreaded reasoning (depending on whether code is inside or outside an atomic block).This paper proposes an alternative strategy that uses yield annotations to control thread interference, and we present an effect system for verifying the correctness of these yield annotations. The effect system guarantees that for any preemptively-scheduled execution of a well-formed program, there is a corresponding cooperative execution with equivalent behavior in which context switches happen only at yield annotations. This effect system enables cooperative reasoning: the programmer can adopt the simplifying assumption of cooperative scheduling, even though the program still executes with preemptive scheduling and/or true concurrency on multicore processors. Unlike bimodal sequential/multithreaded reasoning, cooperative reasoning can be applied to all program code. Keywords Effect system, yield, atomicity, race conditions Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. TLDI'10, January 23, 2010, Madrid, Spain. Copyright c 2010 ACM 978-1-60558-891-9/10/01. . . $10.00Categories and Subject DescriptorsMultiple threads of control are widely used in software development because they help reduce latency and increase throughput on multicore and multiprocessor machines. Reasoning about the behavior and correctness of multithreaded code is difficult, however, due to the need to consider all possible interleavings of the various threads. Thus, methods for specifying and controlling the interference between threads are crucial to the cost-effective development of reliable multithreaded software.Race Freedom Early attempts at controlling thread interference focused on race conditions, where two threads access a shared variable at the same time, and at least one access is a write. Race freedom ensures that thread interference occurs only at certain actions (those that manipulate volatile variables, locks, semaphores, etc). By itself, race freedom does not entirely prevent errors due to unexpected interactions between threads. We illustrate this limitation via the following code, in which the shared variable x is protected by the lock m, and t is a thread-local variable:acquire(m); t = x; release(m); t = t + 1; acquire(m); x = t; release(m); Although this code is race-free, it may not have the expected effect of incrementing m by 1, due to interference from other threads. Similar but less obvious errors arise in practice, often via multiple calls to synchronized methods [19] .Atomicity Motivated by this limitation of race-freedom, much recent work [5, 6, 18-23, 27, 40, 49-52] has used atomic blocks to specify where thread interference may occur. Programs can use traditional synchronization idioms (locks, etc.) to ensure that atomic blocks cannot be influenced by interleaved actions of concurrent threads. The atomic block annotations in a program are correct if any execution trace is serializable, that is, equivalent to a serial trace in which atomic blocks execute contiguously, without interleaved actions of other threads. Atomic blocks are amenable to sequential reasoning, which significantly simplifies subsequent correctness arguments.Despite the benefits of atomicity, it suffers from two significant limitations. First, atomicity forces a form of bimodal sequential/-multithreaded reasoning. To illustrate this point, consider the simple code fragment 1 :x++ If sequential reasoning is valid, then this code is a simple increment operation. If multithreaded reasoning is required, then this code becomes a potentially non-atomic read-modify-write sequence (with no syntactic hints to reflect this sea change from traditional, sequential semantics). Multithreading is increasingly used to improve performance and scalability of programs on today's multicore machines. However, reasoning about multithreaded programs is a difficult endeavor, due to the explosive number of thread interleavings that must be considered for program correctness.. Atomicity is a specification which identifies code blocks that are serializable, that is, equivalent to a serial execution. Although the benefits of atomicity have been explored in prior work, we argue that atomicity has shortcomings which limit its utility. Specifically, atomicity forces two distinct modes of reasoning (sequential/multithreaded) on the programmer, and still exposes the full difficulty of reasoning about concurrency outside of atomic blocks.. We believe that yield annotations offer a complementary approach that provides a single, universal mode of cooperative reasoning throughout the program. This paper introduces a static effect system which guarantees that any preemptive execution of a well-formed program is guaranteed to behave equivalently to some cooperative execution. We hope that our effect system helps to reduce the difficulty of reasoning about multithreaded programs. Lipton [32] proposed reduction as a way to reason about concurrent programs without considering all interleavings. He focused primarily on reasoning about deadlocks. Doeppner [14] , Back [8] , and Lamport and Schneider [31] extended this work to allow proofs of general safety properties.. A number of tools have been developed for detecting race conditions, both statically and dynamically. Our previous work on rccjava [3] uses a type system to catch race conditions in Java programs. This approach has been extended [5, 10, 11] and adapted to other languages [25] . Many other static race detectors have been developed, including Warlock [44] , Locksmith [39] , Chord [36] , and [16, 36, 48] . Eraser [41] detects race conditions dynamically, and this approach has been refined to eliminate false positives, reduce overhead, and handle additional synchronization idioms, as in [37, 46] , or combined with happens-before reasoning [38, 53] . Other approaches have combined dynamic analysis with a global static analysis to improve precision and performance [12, 15, 47] .. A variety of tools have been developed to check for atomicity violations, both statically and dynamically. The Atomizer [18] uses Lipton's theory of reduction [32] to check whether steps of each transaction conform to a pattern guaranteed to be serializable. Wang and Stoller developed an alternative block-based algorithm [51] and also more precise commit-node algorithms [50] . Prior work has developed expressive type and effect systems for verifying atomic specifications in Java programs [5, 19-21, 23, 40, 49] ; we believe these ideas could be adapted to also verify yield annotations.. Our yield effect system was inspired by work on automatic mutual exclusion, a recently proposed concurrency programming model based on having mutual exclusion by default [1, 2, 29] . A key difference is that automatic mutual exclusion enforces a yield annotation at run time, while our effect system uses static analysis to check that yield annotations provide equivalent cooperative execution.. Several recent studies have focused on lightweight transactions [26, 30, 42] and automatic generation of synchronization code from high-level specifications [13, 28, 35, 45] . Much of this work is orthogonal to ours, and while these approaches offer a promising alternative to explicit concurrency control, we believe that a combination of the two approaches will be the most effective programming model for the foreseeable future.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B I O O O O O O B O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B B B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B I O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


When is a function securely computable? A subset of a set of terminals that observe correlated signals seek to compute a given function of the signals using public communication. It is required that the value of the function be kept secret from an eavesdropper with access to the communication. We show that the function is securely computable if and only if its entropy is less than the "aided secret key" capacity of an associated secrecy generation model, for which a single-letter characterization is provided. Suppose that the terminals in M = {1, . . . , m} observe correlated signals, and that a subset A = {1, . . . , a} of them are required to compute "securely" a given (single-letter) function g of all the signals. To this end, following their observations, all the terminals are allowed to communicate interactively over a public noiseless channel of unlimited capacity, with all such communication being observed by all the terminals. The terminals in A seek to compute g in such a manner as to keep its value information theoretically secret from an eavesdropper with access to the public interterminal communication. A typical application arises in a wireless network of colocated sensors which seek to compute a given function of their correlated measurements using public communication that does not give away the value of the function.. Our goal is to characterize necessary and sufficient conditions under which such secure computation is feasible. We formulate a new Shannon theoretic multiterminal source model that addresses the elemental question: When can a function g be computed so that its value is independent of the public communication used in its computation?. In [9] , we had addressed the special case A = M and shown that g is securely computable if and only if its entropy does not exceed the secret key capacity of a standard secrecy generation model [7] , [1] , [5] , [6] . In the general case considered here, a natural extension of this result does not hold, as seen from the Example 1 given in Appendix A, which partly motivates the present work.. We establish that the answer to the question above is innately connected to a new problem of secret key (SK) generation in which all the terminals in M seek to generate "secret common randomness" at the largest rate possible, when the terminals in A c = M/A are provided with side information for limited use, by means of public communication from which an eavesdropper can glean only a negligible amount of information about the SK. The public communication from a terminal can be any function of its own observed signal and of all previous communication. Side information is provided to the terminals in A c in the form of the value of g, and can be used only for recovering the key. Such a key, termed an aided secret key (ASK), constitutes a modification of the original notion of a SK in [7] , [1] , [5] , [6] . The largest rate of such an ASK is the ASK capacity C. Since a securely computable function g for A will yield an ASK (for M) of rate equal to its entropy H, it is clear that g necessarily must satisfy H ≤ C. We show that surprisingly, H < C is a sufficient condition for the existence of a protocol for the secure computation of g for A. When all the terminals in M seek to compute g securely, the corresponding ASK capacity reduces to the standard SK capacity for M [5] , [6] . Furthermore, we show that a function that is securely computed by A can be augmented by residual secret common randomness to yield a SK for A of optimum rate, bringing out an operational decomposition of the entropy in the model.. Preliminaries and the problem formulation are contained in section II. The characterization of the secure computability of g is provided in Section III and a decomposition result for the total entropy of the model is provided in Section IV. The proof of the characterization of the secure computability of g is given in Section V followed by the concluding remarks in Section VI. A full-length version of this submission is currently under review [10] . We obtain simple necessary and sufficient conditions for secure computability involving function entropy and ASK capacity. The latter is the largest rate of a SK for a new model in which side information is provided for use in only the recovery stage of SK generation. This model could be of independent interest. In particular, a function is securely computable if its entropy is less than ASK capacity of an associated secrecy model. The difference is shown to correspond to the maximum achievable rate of an ASK which is independent of the securely computed function and, together with it, forms an ASK of optimum rate. Also, a function that is securely computed by A can be augmented to form a SK for A of maximum rate.. Our results extend to functions defined on a block of symbols of fixed length in an obvious manner by considering larger alphabets composed of supersymbols of such length. However, they do not cover functions of symbols of increasing length (in n), e.g., a running average (in n).. In our proof of Theorem 2, g was securely computed from omniscience at all the terminals in A ⊆M that was attained using noninteractive public communication. However omniscience is not necessary for the secure computation of g, and it is possible to make do with communication of rate less than R CO (A) using an interactive protocol. A related unresolved question is: What is the minimum rate of public communication for secure computation?. A natural generalization of the conditions for secure computability of g by A ⊆M given here entails a characterization of conditions for the secure computability of multiple functions g 1 , ..., g k by subsets A 1 , ..., A k of M, respectively. This unsolved problem, in general, will not permit omniscience for any A i , i = 1, ..., k. For instance with m = 2, A 1 = {1}, A 2 = {2}, and X 1 and X 2 being independent, the functions g i (x i ) = x i , i = 1, 2, are securely computable trivially, but not through omniscience since, in this example, public communication is forbidden for the secure computation of g 1 , g 2 .

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O


Automatic generation of textual summaries from neonatal intensive care data Intensive care is becoming increasingly complex. If mistakes are to be avoided, there is a need for the large amount of clinical data to be presented effectively to the medical staff. Although the most common approach is to present the data graphically, it has been shown that textual summarisation can lead to improved decision making. As the first step in the BabyTalk project, a prototype is being developed which will generate a textual summary of 45 minutes of continuous physiological signals and discrete events (e.g.: equipment settings and drug administration). Its architecture brings together techniques from the different areas of signal analysis, medical reasoning, and natural language generation. Although the current system is still being improved, it is powerful enough to generate meaningful texts containing the most relevant information. This prototype will be extended to summarize several hours of data and to include clinical interpretation. In the Intensive Care Unit (ICU), the interpretation of clinical data is an essential task. Generally, the data available consists of (i) continuously monitored physiological variables (e.g.: heart rate, blood pressure, etc.) and (ii) discrete events (e.g.: equipment settings, drug administration, etc.). However the volumes of data are so large (about 1 MB per patient per day), that attention overload (looking after several patients) and stress can lead to mistakes being be made. Hence, presenting these data in an efficient way is crucial for informed medical decision making.. Although the graphical presentation of data is becoming standard practice, an offward experiment conducted as part of the Neonate project (Hunter et al., 2003) showed that medical professionals, in some circumstances, are more likely to make better treatment decisions if they are given a textual summary of patient data, instead of a graphical one (Law et al., 2005) . In this GraphVsText experiment, forty nurses and doctors with different levels of expertise were asked (individually) to say what actions(s) they would take for a baby whose recent history over a period of about 45 minutes was presented either graphically (as time series plots) or in the form of a textual summary; the text was written by senior clinicians based on the graphical presentation. Each participant was presented with 16 cases, eight graphical and eight textual. Although the clinicians said they preferred the more familiar graphical presentation, they chose more correct actions after reading the textual summaries. Recent experi-mental research comparing textual and graphical summaries of mobile phone manuals (Lagan-Fox et al., 2006) also showed a decision-making improvement with texts.. These results motivated the BabyTalk 1 project whose goal is the automatic generation of texts summarising baby's ICU data in neonatal units. Over the past decade, we have acquired considerable experience in Data Analysis in neonatology with the Cognate (Logie et al., 1997) and Neonate projects as well as in the generation of text from discrete and continuous data with the SunTime (Sripada et al., 2002 , Yu et al. 2007 ) project. A number of techniques have been developed elsewhere for summarising clinical data. For example, the CLEF project (Hallett et al., 2005) aims at generating summaries of multiple text-based health reports. Perhaps the most successful applications have been tools that (partially) automate the process of writing routine documents, such as Hüske-Kraus's Suregen system (Hüske-Kraus, 2003a) , which is regularly used by physicians to create surgical reports; see Hüske-Kraus, 2003b for a review of text generation in medicine. However, the complete summarisation of ICU data is more complex, involving the processing of time series, discrete events, and short free texts, which seems to have not been done before.. The BabyTalk project aims at providing summaries according to two different dimensions: duration and degree of abstraction. Four mains systems are planned: 1. BT-45: descriptive summary of 45 minutes data 2. BT-Nurse: summary of 12 hours of data to serve as a shift summary. 3. BT-Doc: similar to BT-nurse but with the intention of supporting decision making by junior doctors. 4. BT-Family: a daily summary for the baby's family, adapted to the emotional state of the recipient. BT-45 is the simplest system as it covers a limit time period and is purely descriptive. It will be used as a stepping stone to the development of the other systems which involve longer durations and more interpretation. BT-45 will be evaluated by repeating the GraphVsText experiment with three types of presentation: graphical, text written by experts, text generated automatically. This paper describes the progress we have made towards the implementation of BT-45. The data that BT-45 must deal with and the target textual outputs are presented Section 2. The architecture is discussed in Section 3. The prototype has been informally tested by comparison with the manually generated summaries and the results of this comparison are presented in Section 4. The paper ends with a discussion about necessary improvements and other future activities within the project.. Although our work is carried out in the context of neonatal intensive care, we expect the principles to be applicable more widely to adult ICU and to other highdependency units. BT-45 is an ongoing project and there is much to do in order to reach the quality of the experts' text (if indeed this is possible). However, our prototype has demonstrated that it is possible to perform simple data analysis and reasoning that are sufficient to generate a text where the most important information is presented.. Although the Signal Analysis module is composed of simple algorithms which perform in satisfactory way in our scenarios, they must be extended to deal with noisier data. Many artifact removal algorithms exist and an experiment to compare a number of them on noisy neonatal data is planned. Another issue is the detection of human activities from the signal (e.g.: re-siting of probes). For this, an approach based on trend and syntactic analysis will be investigated (Hunter and McIntosh, 1999) .. The Data Abstraction is for the moment very basic. At this level we need to generate more high level abstractions such as the qualification of events (e.g.: "significant" bradycardia), aggregation of similar terms, more linking, etc. This is the weakest component of BT-45 at present and improvements will be based on the acquisition and formalisation of a greater range of expert knowledge.. The way the Content Determination module selects and organizes the information to present in the text is mainly based on the importance factor of the events. This clearly needs to be sensitive to the protocols/procedures being applied and to the characteristics of different babies. Thus, the decision as to which information to hide and to show is an important issue.. The microplanning and realization translates the tree of events into text. It should be more sensitive to aggregation and reference that can be controlled by stylistic parameters (such as desired sentence length). Then, a general lexicalisation engine must be set up to control the usage of technical vocabulary and vague modifiers (such as "small" spike). Moreover, a technique must be implemented to control the multiple time references in the text.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Using disability data to estimate design exclusion Abstract Exclusion auditing is a process that can quantitatively evaluate the inclusive merit of different products, or alternative design decisions. The results from such an audit can provide prioritised directions for product improvement and support the business case for reducing the capability levels required to use mainstream products. The 1996/1997 disability follow-up survey, conducted by the Office of National Statistics, is currently the most comprehensive data source for estimating design exclusion in the UK. In this paper, the data source is explained in detail, and a method is presented that uses this data to estimate the exclusion associated with several tasks that occur in series or in parallel, illustrated through worked examples. Having evaluated how many people are excluded, one can investigate why they were excluded, thus generating design insights into how they could be included. Data from the survey is also converted to a series of stylised graphs, which are intended to inspire designers to think about the relationship between the demands required to use a product and the resulting levels of population exclusion. The Cambridge Engineering Design Centre is unique in developing analytical tools that can quantitatively assess the inclusive merit of different design decisions or products, according to the number of potential users that would be excluded. Such tools have the potential to greatly assist the implementation of inclusive design in businesses [1] . In addition to those excluded from using a product, many more people will experience difficulty or frustration, so reducing the number of people excluded can improve the experience for a wide range of users [2] . Indeed research commissioned by Microsoft [3] reported that 60% of Americans aged 18-64 years were likely or very likely to benefit from the use of accessible technology.. Early work on the psychology of human computer interaction found that the ability to successfully interact with computer terminals could be predicted according to the demands made on sensory, cognitive and motor abilities [4] . Following this work, the ''inclusive design cube'' used sensory, cognitive and motor axes to help represent design exclusion according to various different population definitions [5] . However, more recent psychological models, such as Barnard's Interactive Cognitive Subsystems model [6] , recognise that sensory and cognitive abilities are typically used in combination to perceive information, while sensory, cognitive and motor abilities are all required to perform actions. Figure 1 shows a framework for the typical cycle of perceiving, thinking and acting that occurs for a product interaction, which represents the factors that can predict whether a person will find the use of a product easy, difficult or impossible [7] .. The tools that comprise an exclusion audit intend to present population data about users' context dependent abilities to perceive, think and act, so that designers choosing the interface attributes of mainstream products and services can make better informed decisions to reduce the number of people who find the usage difficult or impossible.. Exclusion auditing is intended to complement other tools for evaluating inclusive merit, such as expert opinion [8] , user trials [9] , and impairment simulators [10] . In combination, these tools provide a holistic approach to discover the causes of design exclusion, and identify appropriate design improvements. An ideal data source for evaluating UK design exclusion would.. • Represent the UK national population, and be customisable towards specific markets of interest.. • Contain data that covers users' abilities to perceive, think and act in real-world contexts.. • Be simple to apply and directly relevant to product interaction.. • Have sufficient scope to incorporate the variation of capability typically found within developed societies, yet sufficient granularity to predict the difference in exclusion between alternative mainstream products.. Unfortunately, no dataset exists that satisfies all of the required criteria [11] . The latest i*design [12] research programme aims to address this shortcoming directly, by designing and undertaking a national survey for the purpose of estimating design exclusion. Until the results from this survey become available, the Office of National Statistics 1996/1997 Disability Follow-up Survey (DFS) [13] remains the best available data source for estimating design exclusion. As a result, this paper focuses on the use of data from the DFS survey. Further details of this survey are first presented, followed by a procedure that uses the survey data to estimate exclusion for one or more tasks, with worked examples. In each case, the objectives are to understand how many people are excluded, why those people are excluded, and provide design insights into how they could be included. The DFS data is then used to construct a set of stylised graphs that communicate how demand relates to exclusion, which are intended to inspire designers to reduce the capability demands of mainstream products, and support the business case for doing so. Further research that is required to inform the future national survey is then discussed. These graphs are intended to present at first glance the relationship between demand and exclusion for each of the different capability categories, in a very general sense. However, the simplified visualisations were constructed through application of artistic licence, and the graphs have very limited accuracy for demand positions other than the four anchor points. Although a true consideration of exclusion arising from multiple capabilities requires using the original DFS capability database, using the set of seven demand and exclusion graphs on their own can provide an order of magnitude approximation for exclusion, and inspire design solutions to reduce it.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O