The Instruction-Set Extension Problem: A Survey The extension of a given instruction-set with specialized instructions has become a common technique used to speed up the execution of applications. By identifying computationally intensive portions of an application to be partitioned in segments of code to execute in software and segments of code to execute in hardware, the execution of an application can be considerably speeded up. Each segment of code implemented in hardware can then be seen as a specialized application-specific instruction extending a given instruction-set. Although a number of approaches exist in literature proposing different methodologies to customize an instructionset, the description of the problem consists only of sporadic comparisons limited to isolated problems. This survey presents a unique detailed description of the problem and provides an exhaustive overview of the research in the past years in instruction-set extension. This article presents a thorough analysis of the issues involved during the customization of an instruction-set by means of a set of specialized applicationspecific instructions. The investigation of the problem covers both instruction generation and instruction selection and different kinds of customizations are analyzed in a great detail. In the past years, electronic devices have been steadily penetrating the market, featuring not only an ubiquitous nature but also a plethora of functionalities. During the years, these functionalities have been implemented by using different kinds of computer architectures which can be categorized according to their degree of flexibility into two main groups: the general-purpose computing group and the application-specific computing group [Bobda 2007; Guo 2006] .. domains. "Additionally, many tools have become available on the market and have allowed programmers to map many different applications onto this type of architectures virtually effortlessly" [Guo 2006 ].. The general-purpose computing group is based on the Von Neumann computing model. The general structure of a Von Neumann machine consists of a memory for holding both program instructions and data (Harvard architectures contain two parallel accessible memories for holding the program instructions and the data separately), a control unit used to store the addresses of the instructions to execute, and an arithmetic and logic unit used to execute the instructions [Bobda 2007] .. A program targeting a Von Neumann machine is coded as a set of instructions to be executed sequentially. The execution of an instruction is realized in five steps: (1) fetching the instruction from the program memory, (2) decoding the instruction to determine which operation has to be executed and which operands are required, (3) reading the operands from the memory, (4) executing the instruction, and (5) writing the result of the operation back to the data memory. This execution model results in a high performance overhead for each individual operation, which turns into energy overhead. In this sense, the general-purpose computing group is considered to be the most flexible hardware at the cost of a general high-energy consumption [Bobda 2007; Guo 2006] .. Over the years, different techniques to increase the level of parallelism have been introduced at instruction level: for instance, techniques such as instruction pipelining, superscalar execution, out-of-order execution, and register renaming. Parallelism has also been exploited at other levels: bit-level, data-level, and loop-level parallelism. Although the level of parallelism has been increased over the years, it is still relatively limited for highly parallelizable applications, which become poor candidates for implementation on these architectures. In this article, we presented an overview of the issues involved in the customization of an instruction-set by means of a set of specialized instructions for a given application or domain of applications. The problems, analyzed in detail, consider different types of customizations and instructions and both instruction generation and selection.. The problems involved, as described in the article, are computational complex problems. Hardware/software partitioning, equivalent to instruction-set customization under certain assumptions, is proven NP-hard in the general case [Arató et al. 2003 ]. Optimal solutions have been proposed by many authors and a plethora of efficient heuristics have been proposed to find near-optimal solutions when the computational complexity of the problem becomes unmanageable and exact solutions cannot be found in a timely manner.. As things stand, one of the major issues in the generation of custom instructions is represented by the degree of human effort required to identify and implement the instruction-set extensions. As described in the article, human ingenuity in manual creation of custom capabilities creates high-quality results. In spite of that, the complexity of the problem as well as the time-to-market requirements led researchers to look for automatic or partially automatic methods for identifying custom instructions. As a result, quality results are produced through a balance of human intervention and automatic methods in the generation of the instructions. However, future approaches will substantially minimize the amount of human effort due to the increasing complexity of the designs.. An additional limitation in the current state-of-the-art in instruction-set extension is the limited number of input and output operands of the custom instructions. This limitation, which is architecture dependent, has been relaxed in the last years by using methods proposed to overcome severe limitations on the number of operands, as mentioned in Section 4.3.3. As a result, new methodologies, by making use of these techniques, will be able to generate and select many more instructions, which in turn will allow a better customization of the instruction-set.. In the last years, many low-power and power-aware architectures have been proposed. While the former minimize power consumption while satisfying performance constraints, the latter maximize performance parameters while satisfying power constraints. The current state-of-the-art in instruction-set customization shows that very few methods exist which take into consideration power issues during the generation of custom instruction. As power consumption/reduction/optimization have become one of the main topics of research, we will see more and more methods appearing for generating custom low-power or power-aware instructions which will be trade off between their size (by limiting the size of the instruction, the power consumption is limited as well) and their frequency of execution (a limited number of executions reduces the power consumption).. One of the main issues in instruction-set customization is also represented by the degree of specialization of the custom instructions. If the instructions are too specialized, instruction reuse becomes hard. This is experienced because it is uncommon that applications from different domains perform the same complex calculations. Vice versa, if a custom instruction is used by many different applications, the requirement to speed up different applications from different domains turns into the generation of custom instructions of limited size. Therefore, the custom instructions are limited to few operations per instruction, which, in turn, can reduce performance. This is a practical issue which is common to every approach and which will be always present: a method for the generation of custom instructions will always be a trade-off between the level of specialization of the instructions and the speed up that they can provide.. Finally, in the last years, multicore systems have become ubiquitous. Many architectures integrate two or more cores in the same hardware to increase performance of execution exploiting the available parallelism. Multicore architectures can provide high performance, run at lower clock speed than single-core architectures, and can reduce power consumption. Multicore systems can be homogeneous or heterogeneous. The former implement identical copies of the same core: same frequencies, cache sizes, functions, etc. Examples are the Intel Core 2 Duo and the Advanced Micro Devices Athlon 64 X 2. Heterogeneous systems integrate different cores which can have different functions, frequencies, memory models, etc. Examples are the CELL Processor used in Sony's PlayStation 3 game console and the Tilera TILE64. Existing methods for the customization of an instruction-set typically consider a single core and a single instruction-set. Nevertheless, in the future, we envision that instruction-set customization will take advantage of the multicore architecture by extending each core with a set of specialized instructions. In this way, a considerable amount of applications from different domains such as graphics, audio, cryptography, communications, mathematics, or biology and more, will be efficiently executed on the architecture.

O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O B O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Finding Bugs in Web Applications Using Dynamic Test Generation and Explicit-State Model Checking Abstract-Web script crashes and malformed dynamically generated webpages are common errors, and they seriously impact the usability of Web applications. Current tools for webpage validation cannot handle the dynamically generated pages that are ubiquitous on today's Internet. We present a dynamic test generation technique for the domain of dynamic Web applications. The technique utilizes both combined concrete and symbolic execution and explicit-state model checking. The technique generates tests automatically, runs the tests capturing logical constraints on inputs, and minimizes the conditions on the inputs to failing tests so that the resulting bug reports are small and useful in finding and fixing the underlying faults. Our tool Apollo implements the technique for the PHP programming language. Apollo generates test inputs for a Web application, monitors the application for crashes, and validates that the output conforms to the HTML specification. This paper presents Apollo's algorithms and implementation, and an experimental evaluation that revealed 673 faults in six PHP Web applications. We have presented a technique for finding faults in PHP Web applications that is based on combined concrete and symbolic execution. The work is novel in several respects. First, the technique not only detects runtime errors but also uses an HTML validator as an oracle to determine situations where malformed HTML is created. Second, we address a number of PHP-specific issues, such as the simulation of interactive user input that occurs when user-interface elements on generated HTML pages are activated, resulting in the execution of additional PHP scripts. Third, we perform an automated analysis to minimize the size of failure-inducing inputs.. We created a tool, Apollo, that implements the analysis. We evaluated Apollo on six open-source PHP web applications. Apollo's test generation strategy achieves over 50 percent line coverage. Apollo found a total of 673 faults in these applications: 72 execution problems and 601 cases of malformed HTML. Finally, Apollo also minimizes the size of failure-inducing inputs: The minimized inputs are up to 5:3Â smaller than the unminimized ones. Adam Kie _ zun received the doctorate degree from the Massachusetts Institute of Technology, doing research on refactoring and on using automated formal-method tools to improve software testing. He is a researcher at Harvard Medical School, working on computational systems biology and human genetics. He previously worked at IBM Research in Switzerland, where he was one of the founding developers of the refactoring engine in the Eclipse Java IDE. Michael D. Ernst is an associate professor in the Computer Science and Engineering Department at the University of Washington. His research aims to make software more reliable, more secure, and easier (and more fun!) to produce. His primary technical interests are in software engineering and related areas, including programming languages, type theory, security, program analysis, bug prediction, testing, and verification. His research combines strong theoretical foundations with realistic experimentation, with an eye to changing the way that software developers work. He was previously a tenured professor at the Massachusetts Institute of Technology, and before that was a researcher at Microsoft Research.. . For more information on this or any other computing topic, please visit our Digital Library at www.computer.org/publications/dlib. The graphs in Figs. 11 and 12 visualize how coverage and the number of failures found increases over time, when both techniques are given up to 20 minutes. For each of the failures reported by the two algorithms, we manually investigated the problem and identified the faulty statements that caused the problem by fixing the problems and making sure that the failures did not recur. Fig. 10 tabulates the final line coverage and faults when both techniques are given 20 minutes. To the best of our knowledge, all reported faults are counted only once.. In most cases, Apollo finds many more failures and achieves much better coverage than Randomized. The only notable exception to this is phpsysinfo (Fig. 12b) , which is not a Web application, but a program that displays system information, and therefore does not rely much on user input. In this case, using the UI analyzer, Randomized was able to perform almost as well as Apollo.. On most applications, Apollo's coverage converges after about 10 minutes while Randomized converges sooner, and at a much lower level. Apollo coverage of webchess (Fig. 11d) 17. Halfond and Orso [20] presented a similar static analysis for Web applications written in Java; however, they discover an approximation of the set of values for each parameter.. 18. Two additional features of the Randomized algorithm are: 1) The user may supply specific values to use for specific parameters (such as username/password), and 2) it detects additional scripts that may be invoked from the output of each execution (user-input simulation).. converges earlier since Apollo fails to login two players, and thus is unable to simulate a chess game. On faqforge (Fig. 11a ), Apollo quickly covers almost 100 percent of the code. In both cases, new failures are still found in the covered code until the last minute of execution. We note that our results confirm the common belief that there is a very strong correlation between coverage and number of failures found.. The Apollo strategy found 673 faults in the subject applications versus only 182 faults for Randomized. Moreover, the Apollo test generation strategy achieved an average line coverage of 52.9 percent versus only 19.8 percent for Randomized.. The coverage of phpbb2 and timeclock is relatively low as the output of these applications contains client-side The #inputs column presents the number of inputs that each strategy created in the given time budget. The coverage column lists the line coverage achieved by the generated inputs. The execution crashes, errors, and warnings and HTML errors and warnings columns list the number of faults in the respective categories. The Total faults columns sums up the number of discovered faults. Figs. 13 and 14 classify the faults reported by Apollo. The execution errors (Fig. 13) are dominated by databaserelated errors, where the application had difficulties accessing the database, resulting in error messages such as 1) "supplied argument is not a valid MySQL result resource" and 2) "Unable to jump to row 0 on MySQL result." The two SQL-related error messages quoted above occurred in faqforge (9 cases of error 1) and webchess (19 cases of error 1 and 1 case of error 2), schoolmate (20 cases of error 1 and 9 cases of error 2), timeclock (1 case of error 1), and phpbb2 (1 case of error 1).. These failures have the same cause: User-supplied input parameters are concatenated directly into SQL query strings and leaving these parameters blank results in malformed SQL, which in turn causes the mysql_query functions to return an invalid result. The subject programs failed to check the return value of mysql_query and simply assume that a valid result is returned. These faults are indications of a potentially serious problem: The concatenation of user-supplied strings into SQL queries makes these programs vulnerable to SQL-injection attacks [12] . Thus, our testing approach indicates possible SQL-injection vulnerabilities despite not being specifically designed to look for security issues.. The three execution crashes (when the interpreter terminates with an exception) in Fig. 10 happen when the interpreter tries to load files or functions that are missing.. For instance, for some inputs that can be supplied to the schoolmate subject program, the PHP interpreter attempts to load a file that does not exist in the current distribution of schoolmate. Since schoolmate has 63 files and PHP is an interpreted language that allows the use of runtime string values when loading files, it is hard to detect such faults. Apollo also discovers a severe fault in the webchess subject program. This fault occurs when the interpreter tries to call to a function that is undefined since the PHP file implementing it is not included due to a value supplied as one of the parameters.. The 673 malformed HTML faults can be divided into several categories (Fig. 14) . These faults are mainly concerned with HTML elements that occur in the wrong place, HTML elements with incorrect values, and with unclosed tags. The breakdown of HTML faults is similar across the different PHP applications. Fig. 13 . Classification of the execution faults found by Apollo. Fig. 14 . Classification of the HTML faults found by Apollo. An earlier version of this paper was presented at ISSTA '08 [2] . The Apollo tool presented there did not handle the problem of automatically simulating user interactions in Web applications. Instead, it relied on a manual transformation of the program under test to enable the exploration of a few selected user inputs. The current paper also extends [2] by providing a more extensive evaluation, which includes two new large Web applications, and by presenting a detailed classification of the faults found by Apollo. In addition, the Apollo tool presented in [2] did not yet support Web server integration.. In the remainder of this section, we discuss three categories of related work: 1) combined concrete and symbolic execution, 2) techniques for input minimization, and 3) testing of Web applications. Next, the algorithm uses a subroutine, getConfigs, to find new configurations. First, the program is executed symbolically on the same input (line 15). The result of symbolic execution is a path constraint, V n i¼1 c i , that is satisfied by the path that was just executed from entry to exit of the whole program. The subroutine then creates new inputs by solving modified versions of the path constraint (lines [16] [17] [18] [19] [20] , as follows: For each prefix of the path constraint, the algorithm negates the last conjunct (line 17). A solution, if it exists, to such an alternative path constraint corresponds to an input that will execute the program along a prefix of the original execution path, and then take the opposite branch, presumably covering new code. The algorithm uses a constraint solver to find a concrete input for each path constraint (line 18).. There are four differences (underlined in the figure) with the simplified algorithm that was previously shown in Fig. 2 . We created a tool called Apollo that implements our technique for PHP. Apollo consists of three major components, Executor, Bug Finder, and Input Generator illustrated in Fig. 7 . This section first provides a high-level overview of the components and then discusses the pragmatics of the implementation.. The inputs to Apollo are the program under test and an initial value for the environment. The environment of a PHP program consists of the database, cookies, and stored session information. The initial environment usually consists of a database populated with some values, and usersupplied information about username/password pairs to be used for database authentication. 9. For simplicity, we omit the details of this function. It compares the username and password to some constants "admin" and "reg.". 10. Attempting to retrieve information from the database using randomly chosen values for username/password is unlikely to be successful. Symbolic execution is equally helpless without the database manager because reversing cryptographic functions is beyond the state of the art for constraint solvers.. The Executor is responsible for executing a PHP script with a given input in a given state. The executor contains two subcomponents:. . The Shadow Interpreter is a PHP interpreter that we have modified to propagate and record path constraints and positional information associated with output. This positional information is used to determine which failures are likely to be symptoms of the same fault. . The State Manager restores the given state of the environment (database, session, and cookies) before the execution and stores the new environment after the execution. The Bug Finder uses an oracle to find HTML failures, stores all bug reports, and finds the minimal conditions on the input parameters for each bug report. The Bug Finder has the following subcomponents:. . The Oracle finds HTML failures in the output of the program. . The Bug Report Repository stores all bug reports found during all executions. . The Input Minimizer finds, for a given bug report, the smallest path constraint on the input parameters that results in inputs inducing the same failure as in the report. The Input Generator implements the algorithm described in Fig. 6 . The Input Generator contains the following subcomponents:. . The UI Option Analyzer analyzes the HTML output of each execution to convert the interactive user options into new inputs to execute. . The Symbolic Driver generates new path constraints from the constraints found during the execution. . The Constraint Solver computes an assignment of values to input parameters that satisfies a given path constraint. . The Value Generator generates values for parameters that are not otherwise constrained, using a combination of random value generation and constant values mined from the program source code. To answer the first research question (Q1) we applied Apollo to the six subject programs and we classified the discovered failures into five groups based on their different failure characteristics:. . Execution crash: The PHP interpreter terminates with an exception.. . Execution error: The PHP interpreter emits an error message that is visible in the generated HTML. . Execution warning: The PHP interpreter emits an error message that is invisible in the generated HTML. . HTML error: The program generates HTML for which the validator produces an error report. . HTML warning: The program generates HTML for which the validator produces a warning report. This classification is a refinement of the one presented in Section 2.3.. To answer the second research question (Q2) we compared both the coverage achieved and the number of faults found with the Randomized generation strategy. Coverage was measured using the line coverage metric, i.e., the ratio of the number of executed lines to the total number of lines with executable PHP code in each application.. We ran each test input-generation strategy for 20 minutes on each subject program, by which time the coverage achieved and number of bugs found by Apollo starts to level off. During this time each strategy generated hundreds of inputs.. This time budget includes all experimental tasks, i.e., program execution, harvesting of constant values from program source, test generation, constraint solving (where applicable), output validation via an oracle, and line coverage measurement. For our experiments, we use the WDG offline HTML validator, version 1.2.2.. We also compared Apollo's results to the results reported by Minamide's static analysis [34] on the four subject programs that we have in common (Section 6.3.1 presents the results).. To answer the third research question, about the effectiveness of the input minimization, we performed the following experiments. Recall that several execution paths and inputs may expose the same failure. Our input minimization algorithm attempts to produce the shortest possible input that exposes each failure. The inputs to the minimizer are the failure found by the algorithm of Fig. 6 along with all the execution paths that expose each failure.

O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B I O O O O O O O O O B B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


River model calibration, from guidelines to operational support tools Numerical modelling is now used routinely to make predictions about the behaviour of environmental systems. Model calibration remains a critical step in the modelling process and different approaches have been taken to develop guidelines to support engineers and scientists in this task. This article reviews currently available guidelines for a river hydraulics modeller by dividing them into three types: on the calibration process, on hydraulic parameters, and on the use of hydraulic simulation codes. The article then presents an integration of selected guidelines within a knowledge-based calibration support system. A prototype called CaRMA-1 (Calibration of River Model Assistant) has been developed for supporting the calibration of models based on a specific 1D code. Two case studies illustrate the ability of the prototype to face operational situations in river hydraulics engineering, for which both data quality and quantity are not sufficient for an optimal calibration. Using CaRMA-1 allows the modeller to achieve the calibration task in accordance with good calibration practice implemented in the knowledge base. Relevant reasoning rules can easily be added to the knowledge base to extend the prototype range of applications. This study thus provides a framework for building operational support tools from various types of existing engineering guidelines. Engineering studies in river hydraulics make extensive use of numerical modelling for various purposes, from environmental applications to flood applications, like flood risk assessment or flood forecasting. But after many years of computational hydraulic practice, model calibration remains a critical and time-consuming task in the commonly defined modelling process. In engineering studies, this process is composed of four main steps: model set-up, model calibration, model validation, and exploitation (Cunge, 2003) . This well-established paradigm has recently faced critics, when physically-based models like river hydraulics models or distributed hydrological models are concerned (Guinot and Gourbesville, 2003) . Critics focus particularly on the way calibration task is commonly undertaken, that is by looking for the most accurate agreement between model outputs and some measured data, often without any -or with few -physical considerations.. In order to define more precisely the position of the calibration task in the modelling process, the present study relies on a framework for terminology in modelling developed by the Society for Computer Simulation (SCS) Technical Committee on Model Credibility (Schlesinger et al., 1979) , and recently extended by Refsgaard and Henriksen (2004) for water-related domains. This framework was modified in order to include the data used during the modelling process and is shown in Figure 1 .. The modified framework is applied to 1-D river hydraulics, where the physical system is a river reach, and the corresponding conceptual model is the Saint-Venant unsteady flow equations. Figure 1 shows that model calibration is only one part of an overall model assessment ). Refsgaard and Henriksen (2004) define the calibration task as "the procedure of adjustment of parameter values of a model to reproduce the response of reality within the range of accuracy specified in the performance criteria", where the performance criteria is the "level of acceptable agreement between model and reality". The first objective of this paper is to clarify this definition in 1D river hydraulics on the basis of heuristic knowledge gained through modelling experience.. Throughout four generations, hydraulic modelling tools changed from basic calculators to powerful, efficient, and versatile tools. With the advent of the third generation, the modelling systems became "tools for building tools" (Abbott, 1991) . In other words, the user was provided with a simulation code and thus only had to perform the model set-up (or model instantiation) and the predictive simulations, together with the corresponding evaluation tasks: model calibration and model validation. But what was pointed out as a "Copernican revolution" in hydraulics by Abbott (1994) was the development and the spread of userfriendly tools with graphical interfaces (Yang et al., 2002) . With the help of the information technology and object-oriented techniques, these hydroinformatics tools allowed more and more engineers to build up their own numerical models. Unfortunately, even modelling packages promoting good modelling practice do not provide significant features to assist users during manual calibration (Dhondia, 2004) . The result is an increasing number of miscalibrated and thus non-predictive models (for illustrative examples, see Cunge et al. (1980) ; Abbott et al. (2003) ).. This situation, along with an increasing demand on an assessment of the credibility of any model, leads to an actual need for calibration support amongst the constantly growing community of hydraulic modellers. This paper presents a framework to transform existing guidelines into an operational support tool, through the development of a knowledge-based calibration support system. The following section describes the different types of guidelines available to hydraulic modellers and the way they are currently disseminated. Section 3 proposes a synthesis of these guidelines in the form of a knowledge base for calibration in 1D river hydraulics, which serves as the core of a prototype calibration support system described in Section 4. Two applications of the prototype are then presented and discussed in Section 5. Assessing the credibility of a simulation is a fundamental issue in modern environmental science ). Good practice is thus required at all stages in the development and evaluation of a numerical model (Jakeman et al., 2006) , and the predictive ability of a model, assessed through model validation, depends largely on the way the model calibration task is performed. This study focuses on the issue of model calibration in the particular domain of 1D river hydraulics and delineates a prototype knowledge-based system to help a modeller achieving this task in accordance with good calibration practice. Throughout the successful application to two different case studies, the prototype knowledgebased calibration support system has proved operational and results on these rather extreme cases are encouraging. Further information about the case studies and the detailed calibration sessions with CaRMA-1 can be found elsewhere (Vidal, 2005) . CaRMA-1 can deal with several calibration features commonly encountered in actual engineering studies, and provides effective support to any modeller, experienced or not, during the calibration task. It has to be emphasized that no prior specific knowledge about river model calibration or about the use of MAGE is required at all to perform a calibration with CaRMA-1. The expertise required is in fact limited to basic hydraulic knowledge to supply the system with site-specific data and to assess agreements between graphical curves.. Among the 25 generic calibration subtasks identified, 18 are performed automatically on the basis of the hydraulic expert knowledge implemented (see Table 1 ). Additionally, as hal-00452243, version 1 -1 Feb 2010 mentioned before, the initialisation of a distributed parameter and the comparison between reference and prediction require the intervention of the user respectively to describe each homogeneous sub-reach and to assess graphical discrepancies. However, both these subtasks are strictly supervised through the use of closed questions.. Knowledge specific to the hydraulic domain is yet to be implemented for 5 The knowledge base currently includes mainly generic and hydraulic knowledge for which a consensus has emerged through the last decades of numerical modelling. Any new reasoning knowledge likely to extend the range of applications of the prototype may easily be added to the knowledge base by adding new concepts (objects) or sub-tasks, or by writing new rules as the ones shown in Table 2 . The system could thus help end-users to solve calibration cases requiring different pieces of empirically-derived reasoning knowledge.. Moreover, some prospects of enhancement of CaRMA-1 have been explored to automate currently interactive parts of the calibration process, and thus free the user from intermediate graphical assessments. Preliminary experiments with a symbolic curve evaluation module have for example been conducted to mimic the way an expert visually identifies discrepancies between reference data and a single model prediction (Vidal and Moisan, 2006 ). This approach is based on fuzzy qualitative descriptions of segments, peaks, and slope breaks of a computed curve, and of deviations from a set of measured points. Such a module would thus allow the knowledge-based system to simulate multi-objective criteria used by experts during visual assessment.

O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O


Robustness-based design optimization under data uncertainty This paper proposes formulations and algorithms for design optimization under both aleatory (i.e., natural or physical variability) and epistemic uncertainty (i.e., imprecise probabilistic information), from the perspective of system robustness. The proposed formulations deal with epistemic uncertainty arising from both sparse and interval data without any assumption about the probability distributions of the random variables. A decoupled approach is proposed in this paper to un-nest the robustness-based design from the analysis of non-design epistemic variables to achieve computational efficiency. The proposed methods are illustrated for the upper stage design problem of a two-stage-toorbit (TSTO) vehicle, where the information on the random design inputs are only available as sparse point and/or interval data. As collecting more data reduces uncertainty but increases cost, the effect of sample size on the optimality and robustness of the solution is also studied. A method is developed to determine the optimal sample size for sparse point data that leads to the solutions of the design problem that are least sensitive to variations in the input random variables.measures are available in the literature: one is the variance, which is extensively discussed in the literature (Du and Chen, 2000; Park, 2001 and Doltsinis and Kang, 2004) and the other is based on the percentile difference (Du et al, 2004) . Although the percentile difference method has the advantage that it contains the information of probability in the tail regions of the performance distribution, this method is only applicable to unimodal distributions. Variance as a measure of variation of the performance function can be applied to any distribution (unimodal or multimodal), but it only characterizes the dispersion around the mean .Feasibility robustness i.e., robustness in the constraints can be defined as satisfying the constraints of the design in the presence of uncertainty. Du and Chen (2000) classified the methods of maintaining feasibility robustness into two categories, methods that use probabilistic and statistical analysis, and methods that do not require them.Among the methods that require probabilistic and statistical analysis, a probabilistic feasibility formulation Lee et al, 2008) , and a moment matching formulation (Parkinson et al, 1993) have been proposed. Du and Chen (2000) used a most probable point (MPP)-based importance sampling method to reduce the computational burden associated with the probabilistic feasibility formulation. The moment matching formulation is a simplified approach which requires only the constraints on the first and second moments of the performance function to be satisfied, and assumes that the performance function is normally distributed. A variation of this approach, the feasible region reduction method has been described in Park et al (2006) , which is more general and does not require the normality assumption. This is a tolerance design method, where In deterministic design optimization, it is generally assumed that all design variables and system variables are precisely known; the influence of natural variability and data uncertainty on the optimality and feasibility of the design is not explicitly considered.. However, real-life engineering problems are non-deterministic, and a deterministic assumption about inputs may lead to infeasibility or poor performance (Sim, 2006) . In recent years, many methods have been developed for design under uncertainty.. Reliability-based design (e.g., Chiralaksanakul 1993; Du and Chen, 2000; Doltsinis and Kang, 2004 and are two directions pursued by these methods. While reliability-based design aims to maintain design feasibility at desired reliability levels, robust design optimization attempts to minimize variability in the system performance due to variations in the inputs (Lee et al, 2008) . In recent years, several methods have also been proposed to integrate these two paradigms of design under uncertainty (e.g., Du et al, 2004 , Lee et al, 2008 ).. Taguchi proposed robust design methods for selecting design variables in a manner that makes the product performance insensitive to variations in the manufacturing process (Taguchi, 1993 ). Taguchi's methods have widespread applications in engineering; however, these methods are implemented through statistical design of experiments and cannot solve problems with multiple measures of performances and design constraints (Wei et al, 2009 ). With the introduction of nonlinear programming to robust design, it has become possible to achieve robustness in both performance and design constraints (Du and Chen, 2000) .. The essential elements of robust design optimization are: (1) maintaining robustness in the objective function (objective robustness); (2) maintaining robustness in the constraints (feasibility robustness); (3) estimating mean and measure of variation (variance) of the performance function; and (4) multi-objective optimization. The rest of this section briefly reviews the literature with respect to these four elements and establishes the motivation for the current study.

O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O