Partnering enhanced-NLP with semantic analysis in support of information extraction Information extraction using Natural Language Processing (NLP) tools focuses on extracting explicitly stated information from textual material. This includes Named Entity Recognition (NER), which produces entities and some of the relationships that may exist among them. Intelligent analysis requires examining the entities in the context of the entire document. While some of the relationships among the recognized entities may be preserved during extraction, the overall context of a document may not be preserved. In order to perform intelligent analysis on the extracted information, we provide an ontology, which describes the domain of the extracted information, in addition to rules that govern the classification and interpretation of added elements. The ontology is at the core of an interactive system that assists analysts with the collection, extraction, organization, analysis and retrieval of information, with the topic of "terrorism financing" as a case study. User interaction provides valuable assistance in assigning meaning to extracted information. The system is designed as a set of tools to provide the user with the flexibility and power to ensure accurate inference. This case study demonstrates the information extraction features as well as the inference power that is supported by the ontology. For a long time, access to relevant documents was a critical impediment for people trying to obtain information on a topic of interest for them. Paper-based documents require the availability of a physical instance of a document, involving the transport of documents with the corresponding costs, delays, and risk factors. Computers and networking infrastructure provide nearly instantaneous access to a huge repository of documents via the World Wide Web and search engines offer support in locating documents that are likely to contain relevant information. The task of examining these documents, extracting relevant pieces of information, and assembling them into a coherent framework, however, still requires significant human involvement, leading to the "information overload" bottleneck. This paper describes the approach our group pursued in the creation of a system that supports humans whose task it is to collect information about a particular domain, analyze that information, assemble it into a coherent framework, and possibly draw conclusions and recommend actions based on knowledge derived from the collected information.. In our context, information extraction (IE) refers to the use of computational methods to identify relevant pieces of information in documents generated for human use, and convert this information into a representation suitable for computer-based storage and processing [Wimalasuriya & Dou, 2010] . IE is often implicitly constrained to text-based documents, although in principle it can be applied to other types such as images, videos or audio recordings as well.. Natural language processing (NLP) refers to the use of computation methods to analyze and process spoken or written statements in a language commonly used by humans. Such methods are applied from different angles. At the syntactic level, grammatical rules are used to determine the basic building blocks of text, such as sentences, words, and the roles they play in a given piece of text. At the semantic level, the meaning of words, phrases, sentences and documents is determined. At the pragmatic level, the context is taken into account as well to determine the most suitable interpretation. Syntactic analysis is relatively straightforward from a computational perspective, but not sufficient to determine the meaning of a text fragment or document; ambiguity, for example, can drastically change the information conveyed in a sentence. Semantic analysis relies on a common interpretation between the creator (writer) and the consumer (reader) of a document. One approach to establish a common interpretation relies on ontologies as frameworks that define the core terminology in a domain and specify the relationships between words. Contextual aspects can be explicitly specified (e.g. through rules), incorporated into a domain-specific ontology, or derived from additional information about the documents and how they are used. Statistical approaches in NLP can overcome this interpretation problem to some degree, and are sometimes combined with the structural analysis methods that rely on rules specifying the grammar of the language.. For humans, speaking the same language is the basis for such a shared interpretation. They use the same principles to construct sentences, although they may not explicitly know the rules of the underlying grammar. For effective communication, they should have a shared vocabulary, meaning that there is a set of words for which the conversation partners have an interpretation, ideally, the same or at least compatible interpretations for one particular word. Contextual information, such as the current location, task, domain of interest, or emotional state of the conversation partner, is considered in cases where multiple interpretations are possible. During the act of reading or listening, humans automatically analyze and interpret language blocks, resolving ambiguities and try to arrive at a coherent interpretation of a document or conversation. Such a holistic treatment of natural language is very challenging for computers. Current systems often combine rule-based approaches, ontologies, and statistical approaches with reasonable success for situations where vocabulary, language structure, or context is constrained. Participants in the Loebner Prize contest, a refinement of the Turing Test, can engage in conversations on limited topics in such a way that it is difficult for an observer to deter-mine if a conversation partner is human or computer [Loebner Prize, 2010] .. Our system aims at the extraction of meaningful pieces of information from wide sets of documents, their integration into a coherent framework, and the derivation of new knowledge. One critical assumption is the existence of such a coherent framework for the domain under consideration. An ontology is a formalized representation of such a framework, and serves multiple purposes in our context. First, it makes explicit the knowledge of humans about the domain. Second, it ensures that the interpretation of critical terms is consistent within the group or organization that utilizes it. Third, it spells out important relationships between those terms. Associated with an ontology can be a set of axioms, which capture the very basic, generally accepted statements about a domain. The flexible use of relations in ontologies allows dynamic, multiple classification of entities. While these properties of ontologies already allow for fairly powerful reasoning, our system also incorporates components for reasoning that are external to the ontology.. On the NLP side, ontologies are the vehicle to provide a semantic framework for the interpretation of sentences and documents, enabling the conversion of statements available in natural language into a representation suitable for computers. For the IE task, an ontology helps in deciding which pieces of information may be relevant, and how they are incorporated into the already existing knowledge repository. The combination of axioms and a flexible hierarchical structure provides a strong basis for reasoning and analysis of the information captured in the repository. Ontologies have a natural visual representation as a graph where nodes represent concepts and links relationships between concepts, and thus serve as a powerful information retrieval method by following interesting relationships.. Ontologies provide major support for several aspects of our system, such as the explicit representation of domain knowledge, interpretation of text, the analysis of documents, and the identification and retrieval of stored information. However, they are difficult and cumbersome to built, may not be available for some areas of interest, and do not capture the full understanding that humans have. The use of ontologies can also become computationally very expensive.. Our overall system is structured as a community of collaborative agents, which share a common objective, but have their own capabilities and the autonomy to make their own decisions. Each agent offers a set of services, and in turn may rely on services provided by other agents.. While we experimented with different domains, our main testing ground is the extraction of information about terrorism financing from publicly available documents on the Web, with news agencies, newspapers, various organizations, and individuals as sources. Clearly the analysis, interpretation, and integration of extracted information is very critical here, and justifies the use of significant human and computational resources. Extracted information can be extremely valuable, and augmenting the capabilities of human analysts may enable them to find overlooked "nuggets" of information, or to draw conclusions they would not have come to otherwise. Obviously the choice of this domain also has some drawbacks: Many relevant documents are not publicly available, and understanding the environment and context in which such analysts work is limited by security constraints [Flynn et al., 2010] . Nevertheless, we believe that the domain of terrorism financing is suitable for the demonstration of our system, and later we will present a usage scenario from this domain.. Computer-based IE and NLP methods have become "good enough" to assist humans with the processing of large quantities of text-based documents. They can identify the occurrence of particular text pieces (words, phrases) in documents, allowing the analysis of simple statements about entities, events, and actions, and the comparison of identified entities against other documents [OpenCalais, 2010] .. The performance of these methods, however, in general is not good enough to leave the task of information extraction completely to computers, especially in fields where the combination of different pieces of information, possibly from a wide range of sub-fields or related areas, is critical. Thus the goal of our project is to demonstrate that computerbased methods for information retrieval, combined with natural language processing and ontologies, enable analysts to deal with significantly larger sets of documents and obtain better results than they would with conventional methods. Our system provides a reusable, configurable platform that can be adapted to specific domains and tasks with moderate effort. It is intended for interactive and collaborative use, and relies on configurable components for document search and retrieval (e.g. Web crawlers), domain-specific language aspects (e.g. rules to identify entities in the domain such as names or specific terms), visual presentation of entities and their relationships (e.g. as topic/concept maps, or ontology visualizations), observation of activities and intermediate results during the extraction and analysis process, and traceability of results to the source documents. The current state of NLP offers some tools to extract information from a collection of documents and recognize basic entity types in that information. In order to perform intelligent analysis on the extracted information, a context has to be provided along with some guidelines for the analysis. We combine existing NLP tools with an OWL ontology to provide a rich environment for performing intelligent analysis. The ontology provides the context and the structure of knowledge in the given domain with the rules, which govern the classification and membership of objects in concepts in the ontology, and provide basic inference mechanism for creating new knowledge. The existing NLP tools extract information from a collection of documents and identify the basic types for entities in the extracted information. Mapping then takes the extracted basic entities and populates the ontology. The inference mechanism runs on the ontology objects and may create new objects or relationships, thus expanding the ontology. The overall system incorporates these tools as collaborative agents that provide and utilize services through a Web server based on Tomcat and Apache. The Web interface provides assistance with the management of sources and documents, and offers support with the extraction, analysis, and retrieval of information contained in and derived from the repository.

O O O B I O O O B I B I O B I I B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B O O O B B O O O O O O O B I O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Quick GPS: A new CAT system for single-part tolerancing This paper depicts a new CAT (Computer Aided Tolerancing) system called Quick GPS (Geometrical Product Specification), for assisting the designer when specifying the functional tolerances of a single part included in a mechanism, without any required complex function analysis. The mechanism assembly is first described through positioning table formalism. In order to create datum reference frames and to respect assembly requirements, an ISO-based 3D tolerancing scheme is then proposed thanks to a set of rules based on geometric patterns and TTRS (Technologically and Topologically Related Surfaces). Since it remains impossible to determine tolerance chains automatically, the designer must impose links between the frames. The CAT system we developed here proposes ISO based tolerance specifications to help ensure compliance with the designer's intentions saving on time and eliminating errors. This paper will detail both the set of tolerancing rules and the designer's approach. The Quick GPS system has been developed in a CATIA V5 environment using CATIA VBA and CATIA CAA procedures. The tolerancing data reported by the designer on technical drawings are becoming nowadays very critical when formalizing the contractual relationships between companies. In industry, functional tolerancing of mechanisms is today more and more based on ISO GPS (Geometrical Product Specification) and ASME Y14.5-2009 standards. The designer's activity consists also on optimizing parts so as to increase mechanism quality while reducing costs. This tolerancing activity requires time, a good knowledge of standards and the use of CAD systems (Computer Aided Design), while the designer must have expertise and skills for selecting appropriate tolerancing specifications. Within a classical approach towards tolerancing, the designer determines the geometrical functional requirements of the mechanism according to function analysis. For each requirement, the designer must identify the influent parts and the 3D tolerance chain, choose tolerances and verify all the results of tolerance chains computation. Current industrial CAD software's do not have specific tolerancing workbenches that cover all the above tasks and only few research works allow quick functional dimensioning and tolerancing based on ISO GPS concepts [1, 2, 3] and tolerance editing controlled by syntactic analysis [4] . FT&A (Functional Tolerancing and Annotation) workbench of CATIA V5 (Dassault Systèmes) [5] uses a specific GD&T (Geometric Dimensioning and Tolerancing) editor and enables syntactic analysis. FT&A is based on TTRS (Technologically and Topologically Related Surfaces) approach [6] . The designer must select datum reference frame and toleranced surface. A dialog box proposes only acceptable specifications and modifiers. Several CAT systems are dedicated to tolerance analysis such as 3DCS ® (Dimensional Control Systems), CETOL ® (Sigmetrix), VSA ® an eM-TolMAte (Siemens PLM). Assembly of parts is simulated (Monte Carlo simulation approach) considering the geometrical defects limited by the chosen tolerances. In Tolerance Manager ® (PCO technologie), the designer must define the set of tolerance chains and the tolerance values. A dashboard presents the results of tolerance chains with worse case or statistical approach. In Mecamaster [7] , the designer creates an isostatic model of the assembly and the system determines the influence of each part deviation on one requirement. SolidWorks DimXpert [8] is used to tolerance a part using either automatic or manual mode based on GD&T Symbols and according to ISO 1101 and ISO 16792 and also ASME Y14.41-2003 standards. Under-constrained, iso-constrained and over-constrained conditions of a part are considered and highlighted by a set of colors. DimXpert GD&T layout forms the basis for Tolerance Analysis using SolidWorks TolAnalyst module. The modular system FROOM (Features and Relations used in Object Oriented Modeling) developed by Salomons and al is based on ACIS ® kernel modeller environment. The first module generates geometrical specifications with TTRS approach [9] . Designer chooses tolerances. The second module performs tolerance analysis with torsor concept [10] . The resulting specifications are very basic and didn't cover for example common zones and modifiers.. In order to provide the designers with complete and reliable solutions, the CLIC system (French acronym for "Cotation en Localisation avec Influence des Contacts") developed by Anselmetti [11, 12, 13] proposes the automatic generating of functional requirements, functional specifications in accordance with ISO tolerancing standards and 3D tolerance chains considering worst case and statistical approaches [14] (Fig. 1) .. Today, CLIC system has been applied successfully and can process simple mechanisms while producing detailed tolerancing data in a short time (10 minutes for a 5 parts mechanism and 100 generated specifications). This paper has demonstrated that a consistent tolerancing of a single part may be achieved with knowledge of both the positioning tables and the links between the various contact surfaces. The Quick GPS method provides an efficient solution for many industries, in cases where each designer is responsible for just a single part. The positioning table offers a way to specify all joints so as to respect positioning requirements. The table summarizes all links into the part and then serves as an alternative to the classical 3D chain transfer approach for the contextual study of a single part. Figure 17 presents body specification results for the positioning requirements. Tolerance values are, by default, set at basic values, e.g. defined at 0.02 for flatness, and must be adjusted by the designer. A|B|C represents the main reference frame (R1b) of the body defined in the body positioning table (Fig. 12) . D|E|F constitutes the auxiliary reference frame for the flange, denoted A1f according to the flange positioning table (Fig. 4) . G represents a portion of the primary auxiliary datum of the shaft, denoted A1s (in Fig. 14) , and requires both orientation and location specifications with respect to the auxiliary reference frame D|E of the flange. The pairs (J,K), (L,M) and (N,P) correspond to the auxiliary reference frames of nipples (A1n, A2n, A3n)..

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Automatic correction of the time step in implicit simulations of the stamping process A speciÿc aspect of simulations of stamping processes is the numerical method adopted to include the non-linearities associated with the frictional contact problem in large deformation plasticity. In fact, convergence problems are frequently encountered, especially when using a quasi-static implicit algorithm. One way of increasing the numerical robustness of these algorithms is to adopt an augmented Lagrangian method to incorporate the problem of contact with friction. The resulting non-linear system can be solved by a Newton -Raphson type method, for which a trial solution must be given to start the iterative equilibrium loop. The quality of this trial solution is determinant for the convergence of the iterative process. An automatic control of the size of the time step can improve this trial solution. In this study, a new size control for the time step, imposing restrictions not only on the increments of the state variables but also on the increments of the contact forces, is proposed. An example of a bending test is used to analyse the in uence of this new limitation on the overall convergence behaviour of the fully implicit code DD3IMP. ? 2004 Elsevier B.V. All rights reserved. The codes that use implicit algorithms to simulate deep drawing processes have general convergence di culties, due to the complexity of the contact problem. However, it should be noted that the accuracy guaranteed for the state variables makes them very promising, encouraging the development of the implicit algorithms to improve their robustness. In the present paper the mechanical model and the numerical methods used in the ÿnite element code DD3IMP 1 , specially developed for the numerical simulation of the three-dimensional deep drawing process, are brie y described. The mechanical model takes into account the large elastoplastic strains and rotations that are associated with the forming process. To include the contact with friction, an augmented Lagrangian method is used, leading to a mixed formulation where the ÿnal unknowns are displacements and contact forces. The obtained non-linear system is solved by a Newton-Raphson method that can exhibit quadratic convergence in the vicinity of the solution [1] [2] [3] .. The global convergence of the Newton-Raphson method, when applied to simulate the stamping process, depends mainly on parameters like: the numerical variables; the mechanical behaviour law; the friction coe cient; the geometry of the tools; the discretization of the metal sheet; the initial solution for the equilibrium iterative loop and the course of the deep drawing process, which can have distinct phases with di erent initial boundary conditions and strain paths. Although it is possible to identify the factors that in uence the convergence, when divergence occurs it is very di cult to identify which of those parameters is the one responsible [3] . The results of the numerical tests undertaken showed that it is always possible to overcome the problems of convergence, acting on numerical parameters. They have also shown that, when divergence occurs, it generally results from di culties in the contact strategy [4] . Modifying the numerical parameters, as the penalty coe cients or the limitations to the size of the time step, has no in uence on the quality of the solution, making implicit methods highly sound and reliable [1] [2] [3] [4] .. In this study, the control of the initial solution of the iterative process is focused. A new set of limitations to the size of the time step is proposed, acting directly on the incremental contact forces that arise from the ÿrst trial solution. Numerical results of the bending of a metal sheet are shown, enhancing the advantages of a time step size controlled also by the trial contact forces. This paper describes the latest improvements on the numerical stability of the ÿnite element code DD3IMP. A general description of the mechanical model and of the numerical methods used in the code is presented in order to depict the new strategy proposed. The code makes use of a fully implicit algorithm of Newton-Raphson type to solve in a single iterative loop the non-linearities associated with the frictional contact and the elastoplastic mechanical behaviour of the material. The use of a Newton-Raphson method guarantees quadratic convergence of the iterative loop as long as the approximated ÿrst solution is not far from the equilibrium. To assure this proximity, control on the time increment is made by user-deÿned parameters. These parameters are usually based on the state variables. A new set of user-deÿned parameters was introduced in order to control the time step in function of the contact force increments.. This set of parameters allows for the overcoming of contact detection and status problems, especially in situations with a large percentage of nodes changing their contact status simultaneously. It is clear from the presented results that the force control parameter cannot, by itself, solve all the convergence problems that can arise during the simulation. Nevertheless, the association of this new strategy with the other set of control parameters improves the numerical stability of the simulations.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Incremental fault diagnosis Abstract-Fault diagnosis is important in improving the circuitdesign process and the manufacturing yield. Diagnosis of today's complex defects is a challenging problem due to the explosion of the underlying solution space with the increasing number of fault locations and fault models. To tackle this complexity, an incremental diagnosis method is proposed. This method captures faulty lines one at a time using the novel linear-time single-fault diagnosis algorithms. To capture complex fault effects, a model-free incremental diagnosis algorithm is outlined, which alleviates the need for an explicit fault model. To demonstrate the applicability of the proposed method, experiments on multiple stuck-at faults, open-interconnects and bridging faults are performed. Extensive results on combinational and full-scan sequential benchmark circuits confirm its resolution and performance.Index Terms-Circuit simulation, fault diagnosis, open-interconnect, very large scale integration (VLSI). T ODAY'S world has been revolutionized by the rapid advancement of very large scale integration (VLSI) integrated circuit (IC) engineering. As the IC process technology becomes more complex and minimum feature size approaches nanometer range, manufacturing quality and yield are becoming more sensitive to physical defects. These defects are usually caused by mask contamination, process variation in fabrication, and spurious material [7] , [10] . Defects can be modeled on the logic level by a fault that affects single or multiple circuit lines, and produces failing output responses for one or more input test vectors. Unraveling the location and cause of the defect, a process known as failure analysis, helps improve the circuit design and manufacturing process leading to a lower cost, improved yield and shorter time-to-market.. Failure analysis usually consists of two tasks: fault diagnosis and defect analysis. Fault diagnosis uses the observed failing responses and the structure of the circuit under diagnosis (CUD) to search for locations that are potentially faulty. This information is used in defect analysis, where the CUD is physically examined to determine and/or eliminate the mechanism of the failure [7] , [10] . Because physical examination is inevitably slow and resource intensive, the efficiency of failure analysis depends on the resolution (i.e., accuracy) of diagnosis. Single-fault diagnosis is a well-studied problem with various linear-time techniques [10] . Although helpful, single-fault diagnosis may not be adequate for defects in modern devices that tend to cluster and affect multiple lines in the failing chip [2] , [3] , [8] , [12] , [13] , [20] , [21] . This is also confirmed in recent experiments from a real-life diagnosis environment, which show that more than 41% of defects found in failing chips cannot be diagnosed using the single stuck-at fault model [8] . Diagnosis of such circuits is a difficult problem because, in theory, the solution space grows exponentially with the number of defective circuit lines [7] , [20] , [24] . Furthermore, a robust diagnosis algorithm should be able to identify defects whose behavior may not be modeled accurately by a fault model [2] - [4] . Clearly, a brute-force approach that exhaustively enumerates all possible locations will need to search a prohibitively large solution space.. To meet these challenges, a simulation-based effect-cause incremental diagnosis approach for clustered defects that affect multiple circuit lines is proposed [17] , [25] . Incremental diagnosis is an iterative process where a single faulty location is identified at each iteration. Fault effects are forced on this line and its fanout cone. These effects can capture either that of a specific fault model (fault-modeled diagnosis) or that of any fault model (model-free diagnosis). Model-free diagnosis is performed by simulating a logic unknown [4] on the candidate fault line. This type of diagnosis is important when the defects' behavior cannot be accurately described by existing fault models [2] , [4] . Incremental diagnosis is repeated until the netlist with forced values emulates the behavior of the faulty chip. Compared to previous incremental approaches [3] , [8] , [12] , [13] , the proposed work differs in the novelty of its algorithms, heuristics, and decision process when it searches the solution space.. To demonstrate the applicability of the proposed work, prototype computer-aided design (CAD) tools are developed for multiple stuck-at fault, open-interconnect fault, and bridging fault diagnosis. We select stuck-at faults because in multiplicity, they can model other types of faults [19] and various types of design errors [24] . Therefore, efficient model-based diagnosis algorithms for multiple stuck-at faults may help find solutions for other fault/error models as well [10] .. Open-interconnect is caused by a break in interconnect wiring due to material or processing defects, electromigration, and thermal stress [5] , [11] . It has been reported that it is the dominant failure in field returns of process chips [22] . Furthermore, the total length of interconnect wires grows rapidly as the International Technology Roadmap for Semiconductors [9] in Fig. 1 , predicts. This trend may increase the failure rates attributed to opens in the near future. Open-interconnect is a fault with a complex behavior [5] , [11] , [19] , [26] . To tackle this complexity model-free incremental diagnosis is used. An extensive suite of experiments using the incremental diagnosis method presented here confirm its efficiency in terms of runtime and resolution. Therefore, it can act as a cost-effective front-end tool to failure analysis to help reduce the number of suspect lines the test engineer has to probe.. This work is organized as follows. The next section describes the motivation and goals of incremental diagnosis. It also outlines the basic components of the algorithm as a sketch for an operating incremental diagnosis framework. In Section III, this framework is tailored to the diagnosis of multiple stuck-at faults (model-based) and multiple open-interconnect (modelfree) faults. Experiments for a variety of faults are found in Section IV and the conclusion follows in Section V. Fault diagnosis is crucial to reduce the cost and time to develop and manufacture IC chips. This work addresses the need for efficient and scalable diagnoses in today's complex VLSI design environment. It proposes an incremental framework for diagnosis that scales well with multiple faults. In this approach, faults are diagnosed one at a time with algorithms that have linear, to the circuit size, performance. Within this framework, diagnosis of two types of faults is examined; stuck-at fault and open interconnect fault. To capture the complex behavior of multiple opens, the idea of model-free incremental diagnosis is introduced. Experiments on circuits for these types of faults but also for faults of different types that are simultaneously present in the circuit confirm the scalability, performance, and resolution of the approach.

O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B B I I I I O O O O O O O O O O O B I I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


A diagrammatic inference system for the web We developed a diagrammatic inference system for the World Wide Web. Our system enables the creation of diagrams such that the information contained in them can be searched and inference can be performed on it. We developed an XMLSchema for bar, line, and pie charts. Based on it, we developed software that transforms a corresponding XML file into an SVG image, which in turn is rendered by the client as an image. Additionally, we developed a search engine which enables a user to find information explicitly contained in the XML file, and as such in the image. Furthermore, we developed an inference engine which enables a user to locate information that is implicitly contained in the image. We developed a process for presenting diagrammatic information on the World Wide Web while making that information available to a search and inference engine. If done right, this makes the information easier to read for people [2] . However, information presented in an image file is not accessible to search engines. While a search engine may be able to use the image tags and surrounding text to make an educated guess about what an image represents, in the case of diagrams (as compared to photos) it is particularly desirable to be able to search and reason with the information contained within the image.. Diagrammatic inference systems are special-purpose systems. They differ from first-order logic in that they are custom-fit for the domain for which they are intended. This means that they are limited to that domain, but they are usually complete [3, 4, 5] . See [1] for an encompassing treatment of these issues. Our system of storing chart information in XML files on the web demonstrates a general process by which diagrammatic information can be stored on a web server in sentential format without losing the preferred visual presentation of the information to the user. The primary benefit of this process is that what the user perceives as an image is in fact a text file that can be indexed by a typical search engine spider. The choice of an XML format reaps the additional benefit of having structured metadata, as demonstrated in the design of our diagrammatic search and inference engine.

O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O B O B O


Fault-scalable Byzantine fault-tolerant services A fault-scalable service can be configured to tolerate increasing numbers of faults without significant decreases in performance. The Query/Update (Q/U) protocol is a new tool that enables construction of fault-scalable Byzantine faulttolerant services. The optimistic quorum-based nature of the Q/U protocol allows it to provide better throughput and fault-scalability than replicated state machines using agreement-based protocols. A prototype service built using the Q/U protocol outperforms the same service built using a popular replicated state machine implementation at all system sizes in experiments that permit an optimistic execution. Moreover, the performance of the Q/U protocol decreases by only 36% as the number of Byzantine faults tolerated increases from one to five, whereas the performance of the replicated state machine decreases by 83%. Today's distributed services face unpredictable network delays, arbitrary failures induced by increasing software complexity, and even malicious behavior and attacks from within compromised components. Like many, we believe it is important to design for these eventualities. In fact, it is increasingly important to design for multiple faults and for graceful degradation in the face of faults. Services are being designed for a massive scale in open clusters (e.g., like the Farsite project [3] ), over the WAN (e.g., like the OceanStore project [34] ), and in administration-free clusters that allow components to fail in place (e.g., like the Collection of Intelligent Bricks project [31] ). In such settings, timing faults (such as those due to network delays and transient network partitions) and failures of multiple components, some of which may be Byzantine in nature (i.e., arbitrary or malicious), may arise. Moreover, rates of server and client crashes in such settings will likely be higher than in carefully-administered settings.. There is a compelling need for services (e.g., namespace, key management, metadata, and LDAP) and distributed data structures (e.g., b-trees, queues, and logs) to be efficient and fault-tolerant. Various services and data structures that efficiently tolerate benign failures have been developed (e.g., Paxos [21] , Boxwood [25] , and Chain Replication [38] ). In the context of tolerating Byzantine faults, the focus has been on building services via replicated state machines using an agreement-based approach [23, 36] . Examples of such systems include the BFT system of Castro and Liskov [9] and the SINTRA system of Cachin and Poritz [8] .. We define a fault-scalable service to be one in which performance degrades gradually, if at all, as more server faults are tolerated. Our experience is that Byzantine fault-tolerant agreement-based approaches are not fault-scalable: their performance drops rapidly as more faults are tolerated because of server-to-server broadcast communication and the requirement that all correct servers process every request.. We have developed the Query/Update (Q/U) protocol as a fault-scalable alternative to these approaches. The Q/U protocol is an efficient, optimistic, quorum-based protocol for building Byzantine fault-tolerant services. Services built with the Q/U protocol are fault-scalable. The Q/U protocol provides an operations-based interface that allows services to be built in a similar manner to replicated state machines [23, 36] . Q/U objects export interfaces comprised of deterministic methods: queries that do not modify objects and updates that do. Like a replicated state machine, this allows objects to export narrow interfaces that limit how faulty clients may modify objects [36] . A client's update of an object is conditioned on the object version last queried by the client. An update succeeds only if the object has not been modified since the client's last query in which case it is retried. Moreover, the Q/U protocol supports multi-object updates that atomically update a set of objects conditioned on their respective object versions.. The Q/U protocol operates correctly in an asynchronous model (i.e., no timing assumptions are necessary for safety), tolerates Byzantine faulty clients, and tolerates Byzantine faulty servers. Queries and updates in the Q/U protocol are strictly serializable [6] . In a benign execution (i.e., an execution in which components act according to specification or crash), queries and updates are obstruction-free [14] . The "cost" of achieving these properties with the Q/U protocol, relative to other approaches, is an increase in the number of required servers: the Q/U protocol requires 5b + 1 servers to tolerate b Byzantine faulty servers, whereas most agreementbased approaches require 3b + 1 servers. Given the observed performance of the Q/U protocol, we view this as a good trade-off -the cost of servers continues to decline, while the cost of service failures does not.. The Q/U protocol achieves its performance and faultscalability through novel integration of a number of techniques. Optimism is enabled by the use of non-destructive updates at versioning servers, which permits operations to efficiently resolve contention and/or failed updates, e.g., by querying earlier object versions and completing partial updates. We leverage this versioning together with a logical timestamping scheme in which each update operation is assigned a timestamp that depends both on the contents of the update and the object state on which it is conditioned. It is thus impossible for a faulty client to submit different updates at the same timestamp-the updates intrinsically have different timestamps-and, so, reaching agreement on the update at a given timestamp is unnecessary. We combine these techniques with quorums and a strategy for accessing them using a preferred quorum per object so as to make server-to-server communication an exceptional case. Finally, we employ efficient cryptographic techniques. Our integration of these techniques has enabled, to our knowledge, the first fault-scalable, Byzantine-resilient implementation for arbitrary services.. We implemented a prototype library for the Q/U protocol. We used this library to build two services: a metadata service that exports NFSv3 metadata methods, and a counter object that exports increment (increment) and fetch (fetch) methods. Measurements of these prototype services support our claims: the prototype Q/U services are efficient and fault-scalable. In contrast, the throughput of BFT [9] , a popular agreement-based Byzantine fault-tolerant replicated state machine implementation, drops sharply as the number of faults tolerated increases. The prototype Q/U-based counter outperforms a similar counter object implemented with BFT at all system sizes in contention-free experiments. More importantly, it is more fault-scalable. Whereas the performance of the Q/U-based counter object decreases by 36% as the number of faults tolerated is increased from one to five, the performance of the BFT-based counter object decreases by 83%. This section overviews the the Q/U protocol for an individual object. Terminology. Q/U objects are replicated at each of the n servers in the system. Q/U objects expose an operationsbased interface of deterministic methods. Read-only methods are called queries, and methods that modify an object's state are called updates. Methods exported by Q/U objects take arguments and return answers. The words "operation" and "request" are used as follows: clients perform operations on an object by issuing requests to a quorum (subset) of servers. A server receives requests; if it accepts a request, it invokes a method on its local object replica.. Each time a server invokes an update method on its copy of the object, a new object version results. The server retains the object version as well as its associated logical timestamp in a version history called the replica history. Servers return replica histories to clients in response to requests.. A client stores replica histories returned by servers in its object history set (OHS ), which is an array of replica histories indexed by server. In an asynchronous system, clients only receive responses from a subset of servers. As such, the OHS represents the client's partial observation of global system state at some point in time. Timestamps in the replica histories in the OHS are referred to as candidates. Candidates are classified to determine which object version a client method should be invoked on at the servers. Note that there is a corresponding object version for every candidate. Figure 2 illustrates candidates, replica histories, and the object history set. Client side. Because of the optimistic nature of the Q/U protocol, some operations require more client steps to complete than others. In an optimistic execution, a client completes queries and updates in a single phase of communication with servers. In a non-optimistic execution, clients perform additional repair phases that deal with failures and contention.. The operations-based interface allows clients to send operations and receive answers from the service; this provides light-weight client-server communication relative to read- ing and writing entire objects. To perform an operation, a client first retrieves the object history set. A client's operation is said to condition on its object history set (the conditioned-on OHS ). The client places the object history set, the method to invoke, and the arguments for the method in a request it sends to servers. By sending the object history set, the client communicates information to the servers about global system state. Both clients and servers classify the conditioned-on OHS to determine which object version a client operation should be performed on at the servers. Classification of a candidate is based on the subset of server replica histories in the object history set in which it appears. (For threshold quorums, classification of a candidate is based on its order, the number of replica histories in the object history set in which it appears.) If all of the replica histories in the conditionedon OHS have the same latest candidate, this candidate is classified as complete. This is illustrated in Figure 2 .. The optimistic nature of the Q/U protocol allows a client to complete queries, and updates conditioned on a cached object history set, in a single phase of client-server communication during failure-and concurrency-free access. Clients cache object history sets. This allows clients to avoid retrieving an object history set before every update operation. So long as no other clients update the object, the cached object history set remains current. Only the quorum of servers that accepted the last update operation have the latest object version. As such, it is most efficient for these servers to process the subsequent update operation. To promote locality of access, and thus efficiency, each object has a preferred quorum at which clients try to access it first. In the case of server failures, clients may have to access a non-preferred quorum. Single phase operations are premised on the client's cached object history set being current (concurrency-free access) and on accessing an object via its preferred quorum (failure-free access).. Server side. Upon receipt of a request, a server first validates the integrity of the conditioned-on OHS. Each server pairs an authenticator with its replica history and clients include the authenticators in the conditioned-on OHS. Authenticators are lists of HMACs that prevent malevolent clients and servers from fabricating replica histories for benign servers. Servers cull replica histories from the conditioned-on OHS for which they cannot validate the authenticator. Authenticators are necessary because servers do not directly exchange replica histories with one another; replica histories are communicated between servers via the client.. If the conditioned-on OHS passes integrity validation, the server performs classification. Next, a server validates that the conditioned-on OHS is current. The server does this by comparing the timestamps of candidates in its replica history with the current timestamp classification identifies for the conditioned-on OHS. If the server has a higher timestamp in its replica history then the current timestamp identified by classification, the conditioned-on OHS is not current. Validating currentness ensures that servers invoke methods on the latest complete object version, rather than one of its predecessors.. If all validation passes, a server accepts the request and invokes the requested method. The method is invoked on the object version corresponding to the latest candidate in the conditioned-on OHS. If the method is an update, a new version of the object results. The timestamp for the resulting object version is a deterministic function of the conditionedon OHS and the operation performed. As such, all servers that receive the same operation and object history set create the same object version and construct the same timestamp for that object version. Timestamps are constructed so that they always increase in value. The server updates its replica history with the resulting timestamp and stores the resulting object version indexed by that timestamp. The server sends a response to the client, indicating success, including the answer to the method, its replica history, and its authenticator. The client updates its object history set with the returned replica histories and authenticators. Once the client receives a quorum of responses that return success, the operation returns successfully. Tolerating failures. If a server crashes, some quorums may become unavailable. This may lead to clients probing additional servers to collect a quorum of responses (probing is the term used to describe finding a live quorum). To protect against malevolent components, timestamps contain the client ID, the operation (method and arguments), and the conditioned-on OHS. Because the operation is tied to the timestamp, an operation can only complete successfully if a client sends the same operation to all servers in a quorum. This mechanism makes it impossible for malevolent clients to force object versions at different benign servers with the same timestamp to differ. The authenticators in the object history set make it impossible for malevolent components to forge replica histories of benign servers; in conjunction with classification rules (see §3.3), this ensures that benign servers only accept requests conditioned on the latest complete object version. Concurrency and repair. Only one update of a specific object version can complete successfully. As such, concurrent accesses of an object may fail and result in replica histories at different servers that have a different latest candidate. In such a situation, a client must bring the servers into a consistent state. To do so, a client performs repair.. Repair consists of a sequence of two special operations: first a barrier is performed, and then a copy is performed. Barrier candidates have no data associated with them and so are safe for servers to accept during periods of contention. Indeed, if contention is indicated by the conditioned-on OHS sent to a server, the server can only accept a barrier candidate. Barrier operations allow clients to safely advance logical time in the face of contention; barriers prevent operations with earlier timestamps from completing. Once sufficient servers have accepted barriers, all contending operations have been suppressed. At this point the client can perform a copy operation. The copy operation copies the latest object version prior to the barrier forward in logical time. Such an object version is either the result of one of the contending operations that triggered repair or the object version these operations conditioned on. In the former case, none of the contending operations was successful, and in the latter case, one of the contending operations was successful. Once a copy operation successfully completes, then it is possible for another update or query to be performed.. Classification of the OHS dictates whether or not repair is necessary and, if so, whether a barrier or copy must be performed. Since both clients and servers base their actions on the OHS, both perform the action dictated by classification.. In the face of contention, clients may have to repeatedly perform barrier and copy operations: different clients could be copying different repairable candidates forward, never completing the copy operation. Client backoff is relied upon to promote progress when there is such contention. Since servers always return their latest replica history to the client, the client can, after updating its object history set, observe the effect of its operation.. Non-preferred quorum access. To be efficient, clients normally access an object via the object's preferred quorum. If a client accesses a non-preferred quorum, some servers that receive requests may not have the conditioned-on object version. In such a scenario, the server must object sync to retrieve the conditioned-on object version. The syncserver requests the needed object data from host-servers. The replica histories in the conditioned-on OHS provides the sync server with information about which other servers are host-servers. Responses from at least b + 1 host-servers are required for the sync-server to validate that the data it receives is correct. The Q/U protocol supports the implementation of arbitrary deterministic services that tolerate the Byzantine failures of clients and servers. The Q/U protocol achieves efficiency by a novel integration of techniques including versioning, quorums, optimism, and efficient use of cryptography. Measurements for a prototype service built using our protocol shows significantly better fault-scalability (performance as the number of faults tolerated increases) in comparison to the same service built using a popular replicated state machine implementation. In fact, in contention-free experiments, its performance is better at every number of faults tolerated: it provides 8% greater throughput when tolerating one Byzantine faulty server, and over four times greater throughput when tolerating five Byzantine faulty servers. Our experience using the Q/U protocol to build and experiment with a Byzantine fault-tolerant NFSv3 metadata service confirms that it is useful for creating substantial services. Much related work is discussed in the course of this paper. This section reviews other Byzantine fault-tolerant protocols that achieve comparable semantics to the Q/U protocol. As such, we do not focus on the extensive prior work implementing Byzantine fault-tolerant read/write registers. Efficient cryptography. Castro and Liskov, in BFT [9] , pioneered the use of lists of HMACs appended to messages as authenticators to implement a replicated state machine. This approach to integrity protection is often achieved via more expensive asymmetric cryptography (i.e., digital signatures). As mentioned in §3.5, the merit of using lists of HMACs for authenticators rather than digital signatures is based on the size of system being evaluated. Byzantine faulty clients. In agreement-based protocols, servers reach agreement on the request sent by a client. Such protocols rely on clients either digitally signing requests or broadcasting their requests to all servers. Previous quorumbased protocols either require an echo phase to ensure that clients prepare the same update at each server in a quorum (e.g., [28] ) or require all servers in a quorum to broadcast messages to all other servers (e.g., [30] ). The Q/U protocol relies on servers retaining object versions ordered by timestamps that include (the hash of) operations and object history sets to protect against Byzantine faulty clients. Agreement-based protocols. There is a long tradition of improving the efficiency of agreement-based protocols (e.g., [7, 33, 18, 9, 8, 20] ). Cachin and Poritz, in the SINTRA project, use a randomized protocol to reach agreement that can be used to build services for WAN environments [8] . The BFT protocol of Castro and Liskov [9] provides comparable semantics and guarantees to the Q/U protocol and is quite responsive (more so than SINTRA). Services implemented with BFT and with the Q/U protocol rely on synchrony to guarantee progress (bounded message delay and backoff respectively), whereas those implemented with SINTRA do not. The public availability, responsiveness, and similarity in guarantees is our rationale for comparing the BFT prototype to the Q/U prototype. Many optimizations make BFT responsive: there is a fast path for read-only operations (like optimistic query execution); BFT reaches agreement on batches of requests to amortize the cost of agreement; and checkpoints, which require digital signatures, are further amortized over multiple batches of requests. Amortizing the cost of digital signatures and agreement over batches of messages was also by Reiter in Rampart [33] and Kihlstrom et al. in SecureRing [18] . Kursawe proposed an optimistic approach that requires one fewer phase of server-to-server broadcast communication, relative to other agreement-based protocols, during failure-free periods [20] . The recent FaB Paxos protocol of Martin and Alvisi also requires one fewer phase of communication, relative to other agreement-based protocols, during stable periods in which correct processes agree upon which correct process is currently the leader (stable periods are more general than failure-free periods) [29] .. Quorum-based protocols. Byzantine quorum systems were introduced by Malkhi and Reiter [26] , and several protocols have been proposed to use them for implementing arbitrary Byzantine-resilient services (e.g., [28, 10, 11] ). To our knowledge, all such protocols are "pessimistic", utilizing multiple phases per update in the common case, and have designs substantially different than that of the Q/U protocol. Our work is the first to demonstrate, in empirical evaluations, a quorum-based protocol for implementing arbitrary Byzantine-resilient services that is competitive with modern implementations of state machine replication at small system sizes, and convincingly outperforms them as the system scales. The Q/U protocol prototype is implemented in C/C++ and runs on both Linux and Mac OS X. Client-server communication is implemented via RPCs over TCP/IP sockets. MD5 cryptographic hashes are employed [35] ; the authors recognize that the MD5 hash is showing its age [39] , however its use facilitates comparisons with other published results for BFT. All experiments are performed on a rack of 76 Intel Pentium 4 2.80 GHz computers, each with 1 GB of memory, and an Intel PRO/1000 NIC. The computers are connected via an HP ProCurve Switch 4140gl with a specified internal maximum bandwidth of 18.3 Gbps (or 35.7 mpps). The computers run Linux kernel 2.6.11.5 (Debian 1:3.3.4-3). Experiments are run for 30 seconds and measurements are taken during the middle 10 seconds; experiments are run 5 times and the mean is reported. The standard deviation for all results reported for the Q/U prototype is less than 3% of the mean (except, as noted, for the contention experiment). The working sets of objects for all experiments fit in memory, and no experiments incur any disk accesses.

O O O B O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O B O O O O O O O O O O O O O B O B O B I I O O O O B O O O O O O O O O O O O O O O B I B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O B I O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I I O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O B O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B B O O O O B O B O O O O O O O B O O O O O O O O O O B I B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O B B O O O O O O O O O O O O O O O O O O B I B O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B I O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Monte

Carlo

method

for

photon

heating

using

temperature-dependent

optical

properties

Powered

by

the

California

Digital

Library

University

of

California

The

Monte

Carlo

method

for

photon

transport

is

often

used

to

predict

the

volumetric

heating

that

an

optical

source

will

induce

inside

a

tissue

or

material.

This

method

relies

on

constant

(with

respect

to

temperature)

optical

properties,

specifically

the

coefficients

of

scattering

and

absorption.

In

reality,

optical

coefficients

are

typically

temperature-dependent,

leading

to

error

in

simulation

results.

The

purpose

of

this

study

is

to

develop

a

method

that

can

incorporate

variable

properties

and

accurately

simulate

systems

where

the

temperature

will

greatly

vary,

such

as

in

the

case

of

laser-thawing

of

frozen

tissues.A

numerical

simulation

was

developed

that

utilizes

the

Monte

Carlo

method

for

photon

transport

to

simulate

the

thermal

response

of

a

system

that

allows

temperature-dependent

optical

and

thermal

properties.

This

was

done

by

combining

traditional

Monte

Carlo

photon

transport

with

a

heat

transfer

simulation

to

provide

a

feedback

loop

that

selects

local

properties

based

on

current

temperatures,

for

each

moment

in

time.

Additionally,

photon

steps

are

segmented

to

accurately

obtain

path

lengths

within

a

homogenous

(but

not

isothermal)

material.

Validation

of

the

simulation

was

done

using

comparisons

to

established

Monte

Carlo

simulations

using

constant

properties,

and

a

comparison

to

the

Beer-Lambert

law

for

temperature-variable

properties.The

simulation

is

able

to

accurately

predict

the

thermal

response

of

a

system

whose

properties

can

vary

with

temperature.

The

difference

in

results

between

variable-property

and

constant

property

methods

for

the

representative

system

of

laser-heated

silicon

can

become

larger

than

K.

This

simulation

will

return

more

accurate

results

of

optical

irradiation

absorption

in

a

material

which

undergoes

a

large

change

in

temperature.

This

increased

accuracy

in

simulated

results

leads

to

better

thermal

predictions

in

living

tissues

and

can

provide

enhanced

planning

and

improved

experimental

and

procedural

outcomes.

Heating

a

medium

using

electromagnetic

irradiation

is

a

common

technique

in

a

variety

of

disciplines,

ranging

from

Corresponding

author.

Tel.:

E-mail

address:

gaguilar@engr.ucr.edu

(G.

Aguilar)..

electronics

to

medicine

to

welding.

There

exist

numerous

applications

where

electromagnetic

heating

is

intended

to

induce

some

thermal

damage

or

structural

change

in

a

medium.

For

example,

in

medicine,

lasers

are

often

used

in

ablative

techniques,

where

the

temperature

of

the

Elsevier

Ireland

Ltd.

All

rights

reserved..

surrounding

tissue

needs

to

stay

within

a

certain

threshold

Other

techniques

use

laser

heating

to

prevent

thermal

damage

inherent

in

cryosurgery

While

lasers

are

often

used

in

medicine

due

to

their

precision,

other

irradiative

techniques

are

gaining

prominence,

such

as

microwave

irradiation

Myriad

tools

currently

exist

to

simulate

the

optical

distribution

within

a

material,

principally

using

the

Beer-Lambert

equation

to

directly

determine

optical

distribution,

or

MonteCarlo

simulations

to

determine

photon

scattering

and

distribution

No

matter

the

method

used,

the

optical

distribution

is

then

used

to

determine

the

thermal

response

of

the

material

to

the

optical

irradiation.

These

approaches

to

simulating

the

optical

distribution

critically

neglect

the

temperature

dependent

nature

of

a

material's

optical

properties..

Current

simulation

techniques

use

constant

optical

properties,

often

at

room

temperature,

to

determine

the

optical

distribution

within

the

material.

While

this

may

in

practice

be

a

reasonable

approximation

for

materials

that

do

not

experience

a

large

change

in

temperature,

this

limitation

becomes

important

as

thermal

gradients

become

large.

This

is

especially

true

in

the

case

of

a

material

which

undergoes

phase

change..

Variable

Optical

Irradiation

Distribution

Simulation

software

(VOIDSim)

was

developed

to

simulate

irradiative

heating

while

using

temperature-dependent

optical

and

thermal

properties.

The

software

not

only

will

determine

the

optical

distribution

within

the

material,

but

also

will

simulate

the

thermal

response

of

the

material

to

the

irradiation.

Furthermore,

the

software

is

robust

enough

to

allow

a

variety

of

irradiative

source

geometries,

temporal

irradiative

profiles,

and

an

arbitrary

system

geometry.

As

exhibited

by

the

small

differences

between

results

of

VOIDSim

and

various

established

methods,

VOIDSim

can

accurately

simulate

photon

propagation.

Furthermore,

it

has

been

demonstrated

that

VOIDSim

has

the

ability

to

correct

significant

error

that

exists

in

a

simulation

where

constant

optical

properties

are

assumed..

For

the

representative

simulation

shown

in

Fig.

both

the

VOIDSim

and

the

Beer-Lambert

law

curves

start

heating

at

the

same

rate,

as

both

absorb

the

same

amount

of

incident

irradiation

initially.

As

the

variable-property

simulation

(VOIDSim)

heats

up,

however,

the

absorption

coefficient

also

increases,

leading

to

greater

absorption

and

faster

heating.

The

two

curves

diverge

as

the

constant-property

simulation

(Beer-Lambert)

will

always

absorb

the

same

percentage

of

incoming

irradiation,

while

the

variable-property

simulation

absorbs

nearly

all

available

irradiation

as

it

warms

up..

There

is

a

difference

of

over

K

between

the

two

curves

after

s,

and

this

difference

will

become

larger

as

time

goes

on.

While

this

temperature

difference

in

a

manufacturing

process

may

be

unimportant,

small

changes

in

temperature

can

be

critical

in

the

heating

of

biological

materials.

As

a

simulation

does

not

currently

exist

that

allows

accurate

prediction

of

system

temperature

where

variable

properties

are

used,

the

capabilities

of

VOIDSim

become

paramount

in

applications

where

optical

properties

vary

with

temperature..

Note

that

a

major

limitation

of

VOIDSim

is

its

inability

to

simulate

temperature-dependent

indices

of

refraction.

This

limitation

was

intentionally

included

due

to

the

high

computational

expense

of

determining

the

optical

path

due

to

continuously

variable

gradients

in

the

index

of

refraction.

Thus

while

domains

may

have

indices

that

vary

from

each

other,

domains

will

always

have

a

homogenous

temperatureindependent

index

of

refraction.

An

additional

limitation

of

VOIDSim

is

that

it

only

considers

solid

and

liquid

phases

of

material,

where

the

liquid

phase

is

locked

in

place

(no

convection).

This

means

that

laser

heating

methods

that

avoid

ablation

may

be

accurately

simulated.

Both

these

limitations

may

be

eliminated

in

future

versions

of

VOIDSim

software..

The

main

advantage

of

VOIDSim

over

open

source

Monte

Carlo

simulators

is

that

it

can

accurately

simulate

photon

absorption

and

the

resulting

heating

and

heat

transfer

in

a

feedback

loop.

This

allows

VOIDSim

to

include

temperature

information

in

determining

the

photon

distribution,

whereas

no

other

simulation

currently

is

able.

As

previously

stated,

this

can

result

in

large

increases

in

accuracy

over

simulations

that

neglect

this

temperature

dependence..

VOIDSim

is

currently

available

as

a

standalone

windows

desktop

application.

As

previously

noted,

VOIDSim

requires

STL

format

CAD

models

to

define

its

geometry.

These

models

may

be

created

within

the

program,

or

can

imported

from

established

solid

modeling

programs.

All

data

entry

and

software

options

are

set

in

a

tabbed

graphical

user

interface.

Rudimentary

data

post

processing

and

visualization

is

done

within

VOIDSim.

Optionally,

VOIDSim

can

automatically

export

data

to

third-party

programs

(i.e.

MATLAB,

SciLab,

Excel)

for

advanced

post

processing

and

visualization.

VOIDSim

software

can

be

installed

in

any

computer

running

Microsoft

Windows,

and

the

speed

of

execution

will

be

highly

dependent

upon

processor

speed

and

RAM.

VOIDSim

is

currently

available

upon

request

from

the

authors..

VOIDSim

was

initially

developed

to

simulate

the

thermal

response

of

frozen

tissue

to

laser

heating.

VOIDSim

allows

for

the

accurate

depiction

of

photon

transport

in

systems

that

undergo

large

changes

in

temperature,

and

hence,

large

changes

in

optical

properties.

Especially

when

considering

living

biological

tissues,

even

small

discrepancies

in

simulated

temperatures

can

literally

mean

life

and

death.

VOIDSim

alleviates

the

discrepancies

introduced

by

the

assumption

of

constant

optical

properties,

potentially

leading

to

better

experimental

and

clinical

planning,

and

more

favorable

outcomes.

Note

that

to

enable

the

commencement

of

preclinical

trials,

a

careful

characterization

of

the

optical

properties

of

tissues

with

regard

to

temperature

is

required,

as

such

data

is

currently

very

sparse

in

the

literature.

B I O O O O O B O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O B O O O O O O O O O O B I B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Synthesizing mood-affected signed messages: Modifications to the parametric synthesis This paper describes the first approach in synthesizing mood-affected signed contents. The research focuses on the modifications applied to a parametric sign language synthesizer (based on phonetic descriptions of the signs). We propose some modifications that will allow for the synthesis of different perceived frames of mind within synthetic signed messages. Three of these proposals focus on modifications to three different signs' phonologic parameters (the hand shape, the movement and the non-hand parameter). The other two proposals focus on the temporal aspect of the synthesis (sign speed and transition duration) and the representation of muscular tension through inverse kinematics procedures. These resulting variations have been evaluated by Spanish deaf signers, who have concluded that our system can generate the same signed message with three different frames of mind, which are correctly identified by Spanish Sign Language signers. Since the last decade there has been an increasing interest in computeranimated signing avatars. Using different techniques, systems have focused on representing signed messages. Currently, all sign language (SL) synthesis efforts are focused on generating neutral signed messages. There is no work 5 that studies how emotional signed messages can be synthesized; this line of research must be established.. To start this line of research it is necessary to answer this question: How are emotions transmitted among humans? Coulson states, "The general conclusion is that emotion, insofar as the term refers to the six 'basic' emotions 10 (anger, disgust, fear, happiness, sadness, and surprise) is accurately perceived through the face and voice [. . . ] Recognition of emotion from posture is comparable to recognition from the voice, and some postures are recognized as well as facial expressions" (Coulson, 2004) . Mehrabian (1981) states that the expression of emotion during communication relies on both visual as-15 pects (appearance, facial expression and body language) and speech-related aspects (mainly the way the voice is used and, to a lesser extent relevance, the message itself). However, how is emotion specifically transmitted through SL? Obviously, its transmission through voice inflexions is not used among deaf people, which leaves facial expressions and body postures as the only 20 ways to transmit this information, according to these authors. Both facial expression and body posture are a part of the phonologic descriptions of signs, and changing a facial expression to represent an emotion, can modify the meaning of a sign. Therefore, the common approach used to represent emotions in general-purpose avatars cannot be directly applied to a signing 25 avatar. Let us return to an interesting point made by Coulson: "emotions are perceived through voice." Speech is the natural way of communicating among those who can hear, and modifications to speech are used to communicate emotions. Sign language is the natural way for the Deaf people to communicate; is it possible to use a similar approach, including some modifi- 30 cations to SL, to communicate paralinguistic information without modifying the message?. The emotional state of the speaker is reflected by means of variations in prosody. Prosody is usually defined as the rhythm, stress and intonation of speech. However, this is not only applicable to speech; it is also applied to 35 signed communication (Sandler and Lillo-Martin, 2006) . In this work, we do not deal with the representation of the basic emotions (Ekman, 1971 (Ekman, , 1999 in signed messages. This work does not attempt to emulate how human signers represent affect. We begin the proposed line of research by focusing on stress in SL (Wilbur and Schick, 1987; Wilbur, 1999) . We present a study of the modifications that can be applied to SL synthesis to represent different degrees of stress without modifying the message: This work pertains to the synthesis of mood-modified signed messages. This work studied the representation of tension in a synthetic signed message. Although it is an important step towards improving synthetic signed messages, it is only the beginning of a line of research. The next step in mood 25 synthesis should focus on emotional synthesis itself by representing the six basic emotions in a synthetic message. This will require both psychological studies about how emotions modify the signed messages and evaluations of the synthetic signed messages that include these modifications. The existing studies on visual prosody and visual speech will serve as the basis in 30 providing an experimental design. The results of this experiment show that it is possible to generate synthetic signed messages representing different stress levels without modifying the meaning of the message. Table 6 shows that the message of the 5 sen-20 tences was kept regarding any modification. This result verifies the condition we imposed: "the message must be the same". This evaluation has been performed by LSE native users. Native users are demanding when it comes to synthetic signed contents and are less tolerant to mistakes than interpreters. The sign recognition rates of deaf user have been reported to be lower than the results obtained when the users who evaluated the system were SL interpreters (López-Colino and Colás, 2011). The acceptance of synthetic signed messages is also lower when evaluated by deaf users (Cox et al., 2003) . Deaf users provide the baseline of the system. When the first four modifications (M 1 -M 4 ) are introduced into the 5 synthetic contents, 74% of the answers correctly identify a relaxed variation and 66% correctly identify a tense variation. None of the users selected the opposite variation. The 'very tense' and 'very relaxed' variations also include the modification M 5 . The results show an increase of the perceived intensity. The results of the sentences which include this modification show 10 that the 82% of the answers correctly identify the variation as 'relaxed' or 'very relaxed' and the 100% of the users consider the variation as being 'tense' or 'very tense'. The represented variation and the users' answers have a significant relationship (χ 2 (16, N = 250) = 403.72, p < .001). These results were not expected considering the results of experiment 15 E 2 . In E 2 , the standard-relaxed comparison was better identified than the standard-tense comparison, contrary to the results of E 3 . We believe that the combination of modifications M 4 and M 5 have created this effect. A gesture involving more muscle effort (M 5 ) performed faster (M 4 ) is clearly perceived as 'tense'. However, this specific aspect of visual prosody has not 20 been studied. The results of the first question of this experiment are presented in Table 5 . The results of the second question reported that each of the 25 pairs of 15 videos showed the same information (see Table 6 ). The validity of our claims was tested through user evaluations, with respect to SL synthesis (Cox et al., 2003; Huenerfauth et al., 2008; Huenerfauth, 2008) and visual prosody (Hartmann et al., 2005a) ; user evaluation is the only approach to test a system or a hypothesis of this kind. The main difference 15 between the two topics is the number of users who participate in the experiments. For studies related to visual prosody, it is usual to find between fifty and one hundred users to perform the experiment, but the evaluations related to SL synthesis are generally performed with six or ten users. This is justified because the people who can perform the second kind of experiments 20 belong to a relatively small social group, namely, the Deaf community. The first three experiments can be performed by anyone, so it was easy to find up to fifty users. The last experiment, which is related to the modified SL synthesis (see section 6) and, required knowledge of SL (in our case, LSE), presented more difficult search for the required users (native LSE signers).

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O