Learning from examples to improve code completion systems The suggestions made by current IDE's code completion features are based exclusively on static type system of the programming language. As a result, often proposals are made which are irrelevant for a particular working context. Also, these suggestions are ordered alphabetically rather than by their relevance in a particular context. In this paper, we present intelligent code completion systems that learn from existing code repositories. We have implemented three such systems, each using the information contained in repositories in a different way. We perform a large-scale quantitative evaluation of these systems, integrate the best performing one into Eclipse, and evaluate the latter also by a user study. Our experiments give evidence that intelligent code completion systems which learn from examples significantly outperform mainstream code completion systems in terms of the relevance of their suggestions and thus have the potential to enhance developers' productivity. The code completion feature of modern integrated development environments (IDEs) is extensively used by developers, up to several times per minute [21] . The reasons for their popularity are manifold. First, usually only a limited number of actions are applicable in a given context. For instance, given a variable of type java.lang.String, the code completion system would only propose members of this class but none of, say, java.util.List. This way, the code completion prevents developers from writing incompilable code by proposing only those actions that are syntactically correct. Second, developers frequently do not know exactly which method to invoke in their current context. Code completion systems like that of Eclipse use pop-up windows to present a Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. list of all possible completions, allowing a developer to browse the proposals and to select the appropriate one from the list. In this case, code completion serves both as a convenient documentation and as an input method for the developer. Another beneficial feature is that code completion encourages developers to use longer, more descriptive method names resulting in more readable and understandable code. Typing long names might be difficult, but code completion speeds up the typing by automating the typing after the developer has typed only a fraction of the name.. However, current mainstream code completion systems are fairly limited. Often, unnecessary and rarely used methods (including those inherited from superclasses high up in the inheritance hierarchy) are recommended. Current code completion systems are especially of little use when suggestions are needed for big (incoherent) classes with a lot of functionality that can be used in many different ways.. For illustration, consider the public interface of the class SWT 1 Text, which consists of more than 160 callable methods. Whenever querying the system for instances of type Text an overwhelming number of proposals are made-including all methods of java.-lang.Object (e.g., equals(), notify(), or wait())! But actually methods like wait() are never invoked on an instance of Text from within the whole Eclipse codebase-after all, a software with several millions of lines of code. Recommending this method every time is rarely helpful to a developer, thus, unnecessarily bloats the code completions and counteracts the (last two) advantages of code completion systems. The method wait() is not an isolated phenomenon. By analyzing the Eclipse codebase, we found that typically no more than five methods are invoked on SWT Text instances. Thus, a developer needs to pick the right 5 method calls from the list of 160 proposals. Even with Eclipse's capability to narrow down the list of proposals with each new character typed by the developer, the list still remains unnecessarily large.. Callable methods differ not only with respect to the frequency of their use; they also often differ with respect to the specific contexts in which they are typically used. Several factors such as the current location in code, e.g., whether the developer is currently working in the control-flow of a framework method [6] , or the availability of certain other variables in the current scope affect the relevance of a method.. For illustration, consider the situation of a developer creating a dialog window to gather user input using a text widget. Typically, text widgets have a two-phase life-cycle: (a) they are configured and placed in a visual container during dialog creation and (b) they are queried for user input after the dialog window is closed. These two phases are typically encoded in different methods of the dialog.. While widget configuration takes place within Dialog.create(), reading the input occurs within Dialog.close(). Depending on the dialog method within which the developer needs suggestions for method calls on a text widget, different methods are relevant. Within Dialog.create(), relevant methods are text widget creation methods and setter methods for its visual properties; within Dialog.close() the getText() method is relevant.. The suggestions made by mainstream code completion systems are based exclusively on the information given by the static type system of the programming language. This seems too primitive given that the examples discussed above suggest that taking into account factors like the frequency of certain method calls in certain contexts might help to improve the accuracy of the proposals made by code completion systems.. Our hypothesis for this paper is as follows: The quality of suggestions and hence the productivity of software development can be improved by employing intelligent code completion systems capable of 1. filtering those elements from the list of proposals which are irrelevant for the current context, thus, disburdening a developer from knowing all (unnecessary) details of the API used, instead allowing him to focus only on API elements that are actually relevant.. 2. assessing the relevance of every proposal (e.g., by using a relevance ranking), thus allowing a developer to quickly decide which recommendations are relevant for the task at hand. In this paper, we propose intelligent code completion systems that learn from existing code repositories by searching for code snippets where a variable of the same type as the variable for which the developer seeks advice is used in a similar context. We have built three prototype code completion engines of this kind, each using the information contained in repositories in different ways. The first prototype uses the frequency of method calls as a metric to decide about their relevance. The second code completion uses association rule mining to search the code repository for frequently occurring method pairs. Finally, the last and most advanced code completion system recommends method calls as a synthesis of the method calls of the closest source snippet found. To build this system, we have modified the k nearest neighbors algorithm [8] , a classical and efficient machine learning algorithm, to fit the needs of code completion. We call the resulting algorithm best matching neighbors (BMN) algorithm.. To evaluate our hypothesis we use a large scale evaluation process for recommender systems, sketched in [6] , along with standard information retrieval performance measures. By large scale, we mean that the system is evaluated with a test bed of more than 27,000 test cases. The evaluation results prove the efficiency of the learning code completion engines in general and of the best matching neighbors (BMN) algorithm in particular. In order to demonstrate that the best matching neighbors (BMN) code completion algorithm, which gets the best quantitative evaluation, has the potential to increase developer productivity, we show how it can be seamlessly integrated into the default Eclipse development widgets and evaluate its usefulness by means of a user study.. The remainder of this paper is organized as follows. In Sec. 2 we present learning algorithms for code completion systems under investigation and especially our main contribution: the best matching neighbors (BMN) algorithm. Sec. 3 presents the setup and results of the quantitative evaluation of the learning code completion systems which are compared against each other and against the default Eclipse code completion system. Sec. 4 presents the prototype implementation of the BMN algorithm and its integration into the Eclipse IDE. Sec. 5 shows the results of a user study that used our tool in real-life situations. In Sec. 6 we give an overview of related work in this area and this paper concludes with Sec. 7. In this paper, we introduced the concept of example based code completion system. These systems are intelligent, their knowledge is based on information mined in an example codebase. They improve the state-of-the-art of code completion systems by producing context-sensitive and relevant method call recommendations, while remaining seamlessly integrated into the IDE.. We presented three example based code completion systems. We conducted a large scale evaluation of these three systems with 27000 real world code completion queries extracted automatically from the example code base. We showed that these systems dramatically outperform the type-based Eclipse code completion system. The best system built is able to predict 82% of the method calls that are actually needed by the programmer (recall) and 72% of the recommended method calls are relevant (precision). It is based on a variant algorithm of the machine learning algorithm K-nearest neighbors, which we called Best Matching Neighbors (BMN).. We performed a user study involving 10 subjects to figure out whether real world developers could benefit from such a new code completion system. The results are promising: 9 out 10 subjects think that an example based code completion system speeds up the development.. There are several areas for future work. Further experiments are needed to gain a more precise understanding of the generalizability of the evaluation results for other kinds of software systems such as classical libraries and standard APIs, as mentioned in Sec.3. Also, there is a lot of machine learning techniques that could leverage the example codebase to build a better method recommendation model. Also, a feedback that we received from the user study was, once you are recommended a relevant method by the system, you still have to figure out yourself how to handle parameters to the call, which is a demanding task. Our future work will address this issue. We believe that it is possible to build a system that recommends what to do with the parameters: using a existing object, instantiating a new one, using a constant value, etc. are all possible alternatives. In the following, we discuss the the generalizability of the evaluation results.. • Our evaluation deals with Eclipse code, i.e. our context (encoded as feature vectors) contains information about the objectoriented context of a variable. For instance, the context might say that the current variable is in a class that extends DialogPage, and in a method that overrides a specific method of DialogPage, say setControl. This additional information may improve the performance of the system, and weaken the generalizability of the very high precision and recall.. • Our evaluation deals with SWT objects. It seems that SWT classes contain more logic, more complex usage patterns than classes of the default Java API (e.g. java.net.URI). Again, the generalizability of the very high precision and recall might be discussable for low-level APIs, such as default Java API. Currently, we are conducting further experiments to assess the performance of the system on different target libraries (without object-oriented contexts, and without complex usage patterns).. • Evaluation queries are built by randomly selecting method calls from real variables of the codebase, without taking into account the order. This may lead to unrealistic queries. For instance, if one considers the ordered calls on a variable, we may randomly drop out the middle third of the calls. In this case, it is unlikely that a developer would write code this way and that the code completion system would be queried with this context. We are in the process of obtaining empirical results to figure out: 1) whether our approximation increases or decreases the overall precision or recall and 2) to what extent the order of method calls matters (for instance, methods that configure an object may be called in different orders). For the questionnaire and interviews we asked the subjects to answer several questions concerning the usefulness of the proposed system and to give their personal rating to these questions. For the personal ratings the subjects had to choose one of Strong Agree (++), Weak Agree (+), Weak Disagree (-), Strong Disagree (--), or No Answer (•) if none of the previous was deemed appropriate. The significant questions from the questionnaire are given below (Questions 2 and 5 are summaries from interviews as they were not rating-based questions).. 1. Did the system propose relevant method calls? (5++ / 4+ / 1 •) This question aims to identify whether subjects' perception was in alignment with the numerical results of the automated evaluation process. Nine out of ten subjects were pleasantly surprised about the quality of the recommendations made by the system but some criticized that in few cases the system also recommended unrelated methods, which lead to deductions in the rating.. 2. Did the system correctly rank the proposals by relevance? (Interview) This question aims to identify how valuable the subjects found the two ways for presenting relevance of a recommendation to the users supported by the system, namely, (a) by presenting the likelihood along with the proposal and (b) by ranking the recommendations by the identified relevance. Most developers stated that they largely ignored the probability of a proposal and just followed the order of the proposals on the code completion window. The interviews suggest that displaying the exact relevance (percentage) is not a necessary feature as long as the most relevant recommendations are found on top of the list.. 3. Did the tool speed up your development compared to the default Eclipse code completion? (4++ / 5+ / 1 -) One reason for the success of code completion is that it speeds up coding. This question aims to catch the subjective perception whether intelligent code completion systems can further improve development speed compared to the default Eclipse code completion. We take the obtained feedback as a positive indicator of the usability of the tool that shows its potential for future research. A large number of framework documentation techniques exists [7] . Prescriptive techniques such as cookbooks, tutorials, exemplars, and pattern languages document how to customize a framework in order to accomplish a specific task [11, 15] . Other work is directed to the documentation of a framework's architecture, so that users understand the rationale behind the design [3] . While helpful, these techniques require that the developer puts much effort to browse and find the good tips concerning the code being written. In contrast, with our approach, the documentation generation is fully automated and integrated into the development environment; the generated documentation is context-sensitively presented to the user by the code completion system.. Code Templates provide a (pre-coded) means of inserting larger code snippets, such as for the creation of an SWT Button. Although code templates are powerful, they suffer from the problem that their creation and maintenance is costly; consequently only few common patterns are encoded using templates. Furthermore, a large number of framework interactions are context-sensitive and consist of single method calls only. For these cases typically no templates exist.. Another category of tools enable users to find code snippets in code repositories. Several tools treat code like regular text documents and apply information retrieval technology to find code examples [10, 12, 17] .. Software exploration tools like Sextant [24] or modern IDEs like Eclipse provide queries to search for code elements based on structural properties. While these search features are better than documentation browsing, they still require that the developer is involved in some heavy interactions with the tool.. Some tools feature implicit queries, i.e., the user is no more required to write a query, but the latter is generated automatically from the current context. For instance, Codebroker [27] uses signature information and comments to actively present similar program elements in its repository. Strathcona [14] uses the structural context of the code under development, e.g., the super type, method calls, or overridden methods to find examples with a high similarity to this context. One disadvantage of example-based tools is, that the developer still has to do the mapping between the recommended code snippet and its own current programming task. Intelligent code completion systems address a different problem. Rather than proposing example code, they use examples to identify common usages of the framework and assess the relevance of method calls found in common usages for the developers current context, giving a developer an intuition which methods are actually relevant and which ones she might want to ignore.. Mylyn [16] shows methods that have been frequently and recently interacted with by the developer at the top of the content assist list. Our approach is complementary. Mylyn learns what to put at the top based on the developer's personal usage history; intelligent code completion systems proposed here learn what to put on top by analyzing code submitted by other developers.. There are three noticeable approaches to help developers when using a framework. They all address the same problem statement: the developer knows what type of object is needed, but not know how to write the code to get the object. Prospector [19] addresses this issue by first creating elementary "jungloids" from framework methods that model the direct reachability of a type tout given a type tin. When the user searches for a type t, the tool follows the jungloids starting from the current code context (all accessible types) until t is reached and returns the code snipped modeled by the synthesized jungloid chain. XSnippet [23] and PARSEWeb [25] extend this approach by improving the ranking heuristics, the query capabilities, and the mining process, and, respectively, by using a code search engine as an example repository which enables to collect code samples on demand. Compared to these approaches, we address a different problem: given an object of some type, what methods can be called on it in a certain context. It is to be noted that Prospector, XSnippet, and Parseweb were evaluated with at most 40 different tasks, while the performance of BMN is assessed by more than 27000 test cases.. Hill and Rideout presented an automatic method completion system [13] which addresses the issue of writing so-called atomic clones. Atomic clones are small scale and good copy/paste of source code. Hill and Rideout argue that atomic cloning is a useful development practice: transforming an atomic clone as a reusable unit is too heavy weight and hinders the design and the extensibility of the software. In order to both help the developer and speed up the writing of an atomic clone, the KNN algorithm is used to match in a code base an atomic clone that is then directly inserted in the source code. While the KNN algorithm and the Euclidean distance matches the problem of searching for atomic clones, we showed why and how to improve the KNN algorithm to create a good code completion system that helps the developers to choose methods to call on an object. Robbes and Lanza [22] showed that it is possible to leverage the program history to improve the quality of code completion. They define and evaluate 6 new algorithms for method name completion. Our approach differs by important points. Evaluation data: Their evaluation strategy requires to have a complete change history of software, obtained by using specific plug-in of the IDE over years of software development. On the contrary, our evaluation requires only to have client source code of the framework or library under study without any special versioning/change data. Note that since we do not have such specific change data, we cannot quantitatively compare our algorithms to theirs. Replicability: The data they used for evaluation is not publicly available. On the contrary, our evaluation is replicable since we use the publicly available Eclipse code-base. Scope: Their algorithms are defined and evaluated in the case where the programmer has already entered at least the two first characters of the method names. Our approach is more ambitious: we aim to predict relevant method names without any prefix entered and proved that it is possible.

O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


NVLab, a Networking Virtual Web-Based Laboratory that Implements Virtualization and Virtual Network Computing Technologies Abstract-In various ICT courses, it is necessary to establish a proper space where each learner has access to a set of network devices with which he/she can build and test networks with different structures and components. This will enable the learners to practice working on multiple devices as in the case of real world context, and freely apply modifications to the network structure to solve any issue that may arise with the current scenario. While on-campus facilities are designed to meet this criterion, current Webbased laboratories either have fixed network designs, or do not offer a number of real devices to work on. Taking a step forward toward building a Web-based laboratory, we used open source Virtualization and Virtual Network Computing technologies in NVLab, a system that offers the learners tools to draw, configure, test, and troubleshoot network designs using real Operating System instances running in the virtual mode on a host machine. The functionality of the system has been experimented by introducing a case study exercise to a group of remote learners as part of Computer Networks course. Evaluating the participants' advancement was done by comparing their answers to a level test prior and after completing the exercise. The evaluation showed that the learners achieved better results in the level test after completing the exercise and were able to use the system efficiently. I N a previous work [4] , we tested the possibility of using Virtualization (Xen) [5] and Virtual Network Computing (VNC) [6] technologies to build a remotely accessible laboratory system that is suitable for Computer Networks courses introducing concepts related to network addressing, routing, traffic flow-controlling, etc. Virtualization technology was used to create a network comprising a number of Virtual Machines (VM) on a single server as shown in Fig. 1 , and in these VMs run real Operating System (OS) instances. With the capabilities modern OSs have, these instances can perform the functions of all network devices such as Routers, IDSs, IPSs, etc. The machines have been set up to enable remote access and learners were able to connect to their consoles through the VNC modules installed on the host machine and on the VMs in order to complete some practical exercises.. Manually creating the VMs and setting them up for remote access involved lot of effort and was a timeconsuming process that instructors were requested to carry out whenever the network was to be rearranged to adopt modifications necessary for introducing new concepts. Furthermore, learners were not able to modify the network structure in order to try scenarios different from the ones the instructors have created. And because certain settings were required for enabling remote access to the VMs, these settings were prohibited from being modified. All these issues brought forth the idea of having a tool for automating the creation of the network, and finding a way to allow learners work on the devices with no restrictions. In this paper, we presented our approach toward building a Web-based Computer Networks laboratory (NVLab) where students can freely practice on designing, configuring, and troubleshooting a network scenario.. To build NVLab, we used Virtualization and VNC as core technologies. Virtualization enables creating a number of networked VMs on one server, and VNC facilitates remote access to the server hosting these VMs in order to configure them and test their work.. The laboratory introduces two new tools for the learners: Designer and Builder. The learners use Designer for producing a network drawing and sending its information as an XML document to be saved on the server side. Then, they will invoke Builder which is responsible for creating Virtual Machines that correspond to those included in the XML document. In the next step, learners will be able to use their Web browsers to connect to the VNC module installed on the host server, start virt-manager (the application through which they can control the VMs), and then start configuring and testing their network.. Responding to our request, a selected group of students with different levels of networking background tried a sample task covering concepts on IP subnetting, connectivity check, and basic routing within NVLab from remote locations.. All the participants could complete the exercise successfully, and the feedback received from them showed that they were satisfied with the way they worked on their designs. The feedback also indicated some notes on the laboratory interface and we will consider working on them in newer versions of the laboratory system. Having verified the appropriateness of the system to be used in computer networks course, we are working on:. 1. Modifying the laboratory structure to accept more than one user logging to the system. 2. Improving Builder to make the creation of the Virtual Machines faster (currently one VM takes 4-5 minutes to be created). 3. Implementing the laboratory in real environment to test it quantitatively.. Muhammad Wannous received the MEng degree in computer science and electrical engineering in 2009 from Kumamoto University, Japan, where he is currently working toward the PhD degree in the Graduate School of Science and Technology. His main research topic is the design of Web-based ICT laboratories and integrating them into the available courses. He has published a number of papers on this topic in international and Japanese conferences. He is a student member of the IEEE and the IEEE Computer Society and is also a member of the IEEE Education Society.. Hiroshi Nakano received the PhD (Doctor of Science) degree from Kyushu University. He is currently a professor in the Center for Multimedia and Information Technology, Kumamoto University. His university teaching experiences are in physics and information science at Nagoya University, and information education and instructional systems (ICT field) at Kumamoto University, where he introduced university-wide e-learning environments and played an important role in founding the Graduated School of Instructional Systems and the Institute of e-Learning Development. His research work focuses on the development of the virtual learning environments, e-laboratory systems and virtual reality for e-learning, and scientific visualization. Work dealing with building a Web-based laboratory environment is of wide range and is not limited to courses related to ICT. But, the design of the laboratory differs according to the field of application and the availability of resources. In this regard, a number of methods have been investigated.. The first approach suggests equipping the on-campus laboratory facilities with remote access network. In this way, learners can access and work on real devices while monitoring the results of the modifications they do to the system. This arrangement has been adopted in courses of wide range such as: electronics engineering [14] , robotics [15] , besides ICT courses [16] , [17] . A drawback of this method is the requirement to have a suitable preinstalled laboratory facility and more resources for the remote access devices. Moreover, changing the configuration of the laboratory requires the presence of some staff on the laboratory's premises or complex switching arrays [18] .. Other researches proposed to use Web-based software simulation tools within the learning environment. Software simulators are rich with animations and variable controls that help learners produce a more meaningful mental model for the introduced concepts. A combination of simulators designed in this way and static materials (text, images, etc.) have proved to be very effective in reaching a better learning performance with medical science students [25] . But, according to a recent research, special care should be given to the complexity of the contents these tools introduce [26] . In other cases, these tools are designed to imitate a hardware device and provide an easy and flexible way of exploring it. In engineering, Web-based laboratories using tools designed in this way are of wide range, [19] in electronics engineering, and [20] , [21] in ICT-related courses. In the software simulators, the laboratory structure can be easily altered and the cost is significantly reduced especially when implementing Open Source code programs. However, a disadvantage of such systems is that work is done on a simulated device which puts limitation on what can be tested since simulators are not designed to cover all aspects of the real device.. The introduction of new technologies for implementing and remotely accessing various devices evoked a new way in designing Web-based laboratories especially in the field of Computer Science. Virtualization technology, for example, is now being used to create several OS instances and run them on one host machine [22] . In parallel, Remote Desktop, Secure Shell, Virtual Network Computing technologies, etc., serve in facilitating access to the laboratory through the Web. Systems implementing these technologies to deliver Computer Networks training have been successfully tested and verified to be adequate to for reaching the goal [23] , [24] . They offer the learners real instances of the laboratory devices to practice on, and feature flexibility and mobility. However, in most cases, these laboratories served to provide training on a narrow range of computer networks concepts such as Network Security, for example. As in all engineering curricula, laboratory activities play a major role in ICT courses. These activities not only help learners get better understanding of what they are being taught, but also provide them with the problem-solving skills and experience necessary for adopting their knowledge in real-life contexts [1] , [2] . Beyond this, laboratory activities became the main components of the course rather than a separate supporting part, and it is now common to find laboratory-based courses delivered at increasing number of universities and training centers [3] .. Among ICT courses that require the learner to work on multiple devices is the Computer Networks course. These days, computer networks have widened to cover the whole world facilitating communication between individuals, access to various resources, e-commerce, etc. That is a reason why well-trained and qualified staff personnel are required at increasing number of companies not only specialized in installation and maintenance, but also in other fields. It is now common to find Network Experts working at stores, government offices, transport stations, and at every place where information is to be exchanged. Consequently, the need for training centers/laboratories utilizing proper infrastructure necessary for qualifying individuals to be Network Experts arises.. Configuring devices operating at different levels of the OSI standard, maintaining connectivity between these devices on one hand and with the external world on the other hand, traffic and load balancing, and upholding carefully designed security policy are among the network administration tasks. Hence, a laboratory addressing Computer Networks issues should contain enough machines of various types, computers, switches, routers, firewalls, IDSs, IPSs, etc., to cover all the above mentioned tasks, among others. Besides, every individual learner should have access to a set of these devices in order to try different possible network scenarios that meet the requirements and pick one of them to be implemented.. While on-campus facilities are designed to meet the previously mentioned criterion, Web-based laboratories are evolving to become equal. This has been reinforced by the introduction of emerging technologies and the increased demand for them to be used within the e-learning environments. Systems implementing emerging Virtualization and Remote-Access technologies to deliver Computer Networks training have been successfully tested and verified to be adequate for reaching the goal [23] , [24] . The need to supply each learner with a Web-based space offering a set of devices to try and test different network scenarios was the main reason behind introducing NVLab. Further than this, it is required to have the proper tools that enable learners construct and access the networks they want to test. In the next part, we describe how Xen and VNC were implemented, and the different tools provided to the learners in the suggested system.

O O O O O O O O B O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Wireless sensor networking for rain-fed farming decision support Wireless sensor networks (WSNs) can be a valuable decision-support tool for farmers. This motivated our deployment of a WSN system to support rain-fed agriculture in India. We defined promising use cases and resolved technical challenges throughout a two-year deployment of our COMMON-Sense Net system, which provided farmers with environment data. However, the direct use of this technology in the field did not foster the expected participation of the population. This made it difficult to develop the intended decision-support system. Based on this experience, we take the following position in this paper: currently, the deployment of WSN technology in developing regions is more likely to be effective if it targets scientists and technical personnel as users, rather than the farmers themselves. We base this claim on the lessons learned from the COMMON-Sense system deployment and the results of an extensive user experiment with agriculture scientists, which we describe in this paper. Today emerging countries face the antagonistic goals of developing their economies and improving their population's livelihood, while sustaining their natural resources and environment. Environmental monitoring seems to hold great promise for tackling these issues. One typical example is air pollution monitoring, because this is a major concern in many urban areas of developing countries, especially in the mega cities of Asia. Water quality control is another opportunity: in developing countries, diarrhoeal diseases consecutive to the consumption of contaminated water are one of the main causes of child mortality.. Most of all, agriculture comes as a natural application, given the importance that climatic and physical parameters have throughout the development of the crop. Indeed, developing countries often have to cope with a large farming population who have seen their situation deteriorate in recent years due to the instability of market prices and the perceived effects of climate change.. As a typical example, the share of agriculture for employment in India is about 67%, with a majority of small land holdings. In Karnataka (Southern India), 87% of the farms have less than 4 ha and account for more than 50% of the total cultivated land, while 39% of the total are very small farms (less than 1 ha). These resourcepoor farmers usually lack access to irrigation facilities and depend on rain-fed farming for their livelihood [1] . Their crop yields are highly unreliable, due to the variability in both the total rainfall and its distribution [2, 3] . Unlike industrial farming companies, they face daunting challenges, as is illustrated by the current wave of suicides throughout the country [4] .. We performed a survey in 2004 to identify the user needs of the rural population in three small villages of Karnataka [5] : the themes of crop yield prediction, pest and disease control, and water levels in bore wells stood out prominently among farmers' responses. Practicing exclusively rain-fed farming, they would greatly benefit from a better knowledge of the field environment, and from more adapted strategies and practices.. To address these issues, it seems natural to assess the potential of wireless sensor networks (WSNs), in particular, the increased space and time resolution they bring to environmental data at an affordable price. WSNs are today viewed as an enabling technology for precision agriculture: irrigation management, frost prevention, nutrient or pesticides control are documented examples [6] . Still, it remained to be proven whether a similar knowledge could be used by the population who needs it most: the resource-poor farmers who practice rain-fed agriculture.. This has been the endeavor of the COMMON-Sense Net [8] project, which has investigated over three years the utilization of WSNs as a decision-support tool for rain-fed farming. Our goal was to help the farmers monitoring the physical farming environment, in order to understand more precisely the physical processes at hand, and to react optimally to changing conditions. Our vision has been to bring the benefits of technology directly to farmers, in a participatory way. We deployed our system, described in Sec. 2, after identifying use cases with the locals. Over a long period of data collection and usage, we reached an impasse: numerous difficulties have emerged, essentially hindering our efforts to bring in a participatory manner the added value of WSN technology, through our system, to the farmers.. This has been primarily due to the farmers' alienation with the worlds of science and technology. Presenting the lessons learned through our three-year long effort and deployment, in Sec. 3, we are currently forced to consider as somewhat idealistic our initial objective. The resource-poor farmers could not really put enhanced environmental data in use effectively.. Based on these experiences, we moved on with a different approach: we investigate controlled-environment strategies related to rain-fed farming, such as developing new crop varieties or pest prediction measures. Accordingly, the position this paper takes is as follows: Under the current conditions in developing regions, such as Karnataka, the targeted users for WSN-enabled applications, should be researchers, scientists, and technicians. As our study case, long experience, and recent experiment we conducted in Bangalore, from November 2007 to February 2008, indicate, this orientation towards a new user group (of scientists) that can then advise or guide the farmers appears currently the only method to have an effective decision-support system for rain-fed farming.. This article brings this seemingly pessimistic yet realistic position to the attention of the community. Essentially, our concerted effort of deploying and running the COMMON-Sense Net system points that still scientists remain the preferred 'customers' of WSN technology. We present the learned lessons that led us to this shift away from our initial objective. Then, Sec. 4 describes the methodology for our user experiment to determine if our new user target group was an appropriate choice. In Sec. 5, we present our results, followed by a discussion, future work and conclusions. We performed an experiment about the usefulness and usability of wireless sensor networks for rain-fed agriculture. The usefulness to provide finer-grained environmental data for applied research in this field was clearly identified. The technology gap remains, however, and can only be bridged with a highly dynamic process of interaction with potential users. Building appropriate partnerships and exploring the outcomes of medium and long-term deployments of WSNs is the next step. The interest from the non-governmental sector is the most promising because it presents the most direct contact with the intended beneficiaries of this technology: the small land-holding farmers. Potential users expressed keen interest in several cases. In particular, a major NGO working in the field of dry-land farming throughout India expressed interest in the WSN technology. Such promising results must be tempered by the low response obtained by the application use, which we address in section 6.3.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Selecting a Cost-Effective Test Case Prioritization Technique Regression testing is an expensive testing process used to validate modified software and detect whether new faults have been introduced into previously tested code. To reduce the cost of regression testing, software testers may prioritize their test cases so that those which are more important, by some measure, are run earlier in the regression testing process. One goal of prioritization is to increase a test suite's rate of fault detection. Previous empirical studies have shown that several prioritization techniques can significantly improve rate of fault detection, but these studies have also shown that the effectiveness of these techniques varies considerably across various attributes of the program, test suites, and modifications being considered. This variation makes it difficult for a practitioner to choose an appropriate prioritization technique for a given testing scenario. To address this problem, we analyze the fault detection rates that result from applying several different prioritization techniques to several programs and modified versions. The results of our analyses provide insights into which types of prioritization techniques are and are not appropriate under specific testing scenarios, and the conditions under which they are or are not appropriate. Our analysis approach can also be used by other researchers or practitioners to determine the prioritization techniques appropriate to other workloads. As software evolves, test engineers regression test it to validate new features and detect whether new faults have been introduced into previously tested code. Regression testing is important, but also expensive, so many approaches for improving its costeffectiveness have been investigated. Among these approaches we find test case prioritization. Test case prioritization techniques help engineers execute regression tests in an order that achieves testing objectives earlier in the testing process. One testing objective involves rate of fault detection-a measure of how quickly a test order detects faults. An improved rate of fault detection can provide earlier feedback on the system under test, enable earlier debugging, and increase the likelihood that, if testing is prematurely halted, those test cases that offer the greatest fault detection ability in the available testing time will have been executed.. Numerous prioritization techniques have been described in the research literature (Elbaum et al., 2001b Jones and Harrold, 2001; Rothermel et al., 2001; Wong et al., 1997) . Studies Rothermel et al., 1999 Rothermel et al., , 2001 have shown that at least some of these techniques can significantly increase the rate of fault detection of test suites in comparison to the rates achieved by unordered or randomly ordered test suites. More recently, researchers at Microsoft (Srivastava and Thiagarajan, 2002) have applied prioritization to test suites for several multi-million line software systems, and found it highly efficient even on such large systems.. These early indications of potential are encouraging; however, studies have also shown that the rates of fault detection produced by prioritization techniques can vary significantly with several factors related to program attributes, change attributes, and test suite characteristics (Elbaum et al., 2001a (Elbaum et al., , 2003 . In several instances, techniques have not performed as expected. For example, one might expect that techniques that take into account the location of code changes would outperform techniques that simply consider test coverage without taking changes into account, and this expectation is implicit in the technique implemented at Microsoft (Srivastava and Thiagarajan, 2002) . In our empirical studies , however, we have often observed results contrary to this expectation. It is possible that engineers choosing to prioritize for both coverage and change attributes may actually achieve poorer rates of fault detection than if they prioritized just for coverage, or did not prioritize at all.. More generally, to use prioritization cost-effectively, practitioners must be able to assess which prioritization techniques are likely to be most effective in their particular testing scenarios, i.e., given their particular programs, test cases, and modifications. Toward this end, we might seek an algorithm which, given various metrics about programs, modifications, and test suites, calculates and recommends the technique most likely to succeed. The factors affecting prioritization success are, however, complex, and interact in complex ways (Elbaum et al., 2001a) . We do not possess sufficient empirical data to allow creation of such a general prediction algorithm, and the complexities of gathering such data are such that it may be years before it can be available. Moreover, even if we possessed a general prediction algorithm capable of distinguishing between existing prioritization techniques, such an algorithm might not extend to additional techniques that may be created.. In this paper, therefore, we pursue an alternative approach. Using data obtained from the application of several prioritization techniques to several substantial programs, we compare the performance of these prioritization techniques in terms of effectiveness, and show how the results of this comparison can be used, together with cost-benefit threshold information, to select a technique that is most likely to be cost-effective. We then show how an analysis strategy based on classification trees can be incorporated into this approach and used to improve the likelihood of selecting the most cost-effective technique.. Our results provide insight into the tradeoffs between techniques, and the conditions underlying those tradeoffs, relative to the programs, test suites, and modified programs that we examine. If these results generalize to other workloads, they could guide the informed selection of techniques by practitioners. More generally, however, the analysis strategy we use demonstrably improves the prioritization technique selection process, and can be used by researchers or practitioners to evaluate techniques in a manner appropriate to their own testing scenarios.. The rest of this article is organized as follows. Section 2 describes the test case prioritization problem in greater detail, presents a measure for assessing rate of fault detection and techniques for prioritizing test cases, and summarizes related work. Sec-tion 3 presents the details of our study, our results, and our approaches for selecting appropriate techniques. Section 4 presents further discussion of our results, and concludes. Rothermel et al. (2001) define the test case prioritization problem and describe several issues relevant to its solution; this section reviews the portions of that material that are necessary to understand this article. We have presented a study of five test case prioritization techniques applied across eight systems. Although previous studies of test case prioritization (Elbaum et al., 2001b Rothermel et al., 1999 Rothermel et al., , 2001 ) have been conducted, the set of subjects we have considered form the largest set of non-trivial programs employed to date to quantify the relative effectiveness of prioritization techniques at improving the rate of fault detection of test suites. 3 Our results regarding the effectiveness of the techniques confirm previous findings (Elbaum et al., 2001b Rothermel et al., 1999 Rothermel et al., , 2001 , among them the fact that the performance of test case prioritization techniques varies significantly with program attributes, change attributes, test suite characteristics, and their interaction.. These results support the search for strategies by which practitioners could choose appropriate prioritization techniques for their particular testing scenarios, and we have proposed two such strategies. The basic instance-and-threshold strategy, introduced in Section 3.3.1, recommends the technique that has been successful in the largest proportion of instances in the past, accounting for cost-benefit thresholds. The enhanced instance-and-threshold strategy, introduced in Section 3.3.2, adds into consideration the attributes of a particular testing scenario, using metrics to characterize scenarios, and employing classification trees to improve the likelihood of recommending the proper technique for each particular case.. The relative effectiveness of these two strategies for a cost-benefit threshold of 1% is summarized in Table 10 . Each row introduces the techniques being compared, the probability for recommending the appropriate technique for a given scenario under each strategy, and the gain generated by the enhanced strategy with respect to the basic strategy. For example, in the first row, we see that a practitioner employing the basic instance-and-threshold strategy would have a 61% likelihood of selecting the most effective technique. A practitioner using the enhanced strategy, however, would have an 88% likelihood of selecting the most effective strategy (at the cost of computing AP_FET and following the classification tree introduced in the previous section).. The effectiveness of these strategies on the workloads we considered demonstrates their viability for evaluating techniques in other scenarios introduced by researchers or practitioners. In particular, researchers or practitioners wishing to evaluate new techniques or scenarios should begin by considering the basic instance-and-threshold strategy, because this simpler approach may be sufficient for their particular scenarios. In the cases in which this approach does not lead to clear technique selections, and the enhanced strategy seems desirable, the process illustrated in this paper can be followed. It is best to begin, however, with simple, inexpensive metrics that seem likely to characterize the scenarios, and refine these metrics if gains achieved by classification trees are limited. In any case, the more data that is accumulated for use in classification, the better the result is likely to be. In this work we have assumed that the prioritization techniques examined have equivalent costs. For the relatively simple techniques we have considered, all operating at the level of function coverage and using binary "diff" decisions that could be retrieved from configuration management, this assumption seems reasonable. When seeking to extend these comparisons to other classes of techniques, however, this assumption would be less reasonable. Techniques that incorporate test cost or module criticality information, or those that operate at finer grained levels of coverage, present different cost-benefits tradeoffs. These tradeoffs can be modelled as described in , and related to cost-benefit thresholds, allowing comparisons of differing-cost techniques, but this approach needs to be investigated empirically.. The most pressing need for future work, however, involves additional studies of prioritization applied to a wider variety of programs, modified programs, test suites, and faults. Such studies could help us extend the generality of our conclusions about relative technique efficacy. Further data will also help us better understand the characteristics influencing cost-effectiveness, and help us further differentiate techniques through additional metrics and the classification tree approach. The need for such studies is particularly acute because software professionals are beginning to employ test case prioritization in practice (Srivastava and Thiagarajan, 2002) , and our results show that some of the expectations we might have about technique effectiveness (e.g., "adding modification information will improve rate of fault detection") can be incorrect. Practitioners relying on such expectations may actually, and unwittingly, be harming their test suites' rates of fault detection. Only through careful empirical work can we transform such unfounded and potentially problematic expectations into useful engineering strategies. In recent years, several researchers have addressed the test case prioritization problem and presented techniques for addressing it. Wong et al. (1997) suggest prioritizing test cases according to the criterion of "increasing cost per additional coverage." The authors restrict their attention to prioritization of the subset of test cases selected from a test suite by a safe regression test selection technique, and the selected test cases are only those test cases that reach modified code, but other test cases could be placed after these for later execution. Thus, this technique can be described as using feedback, modification information, and test cost information. Rothermel et al. (1999 Rothermel et al. ( , 2001 and Elbaum et al. (2001b Elbaum et al. ( , 2002 , in work summarized above, provide the first formal definition of the prioritization problem and present metrics for measuring the rate of fault detection of test suites. They define prioritization techniques including all of those described in Section 3.1, and present the results of several empirical studies of those techniques. Jones et al. (2001) describe a technique for prioritizing test cases for use with the modified condition/decision coverage (MCDC) criteria, this technique uses feedback, but no modification information. Srivastava and Thiagarajan (2002) present a technique for prioritizing test cases based on basic block coverage, using both feedback and change information. The technique also differs from previous techniques in that it computes flow graphs and coverage from binaries, and attempts to predict possible affects on control flow follow-ing from code modifications. The authors describe the application of this technique to several large systems at Microsoft, and provide data showing that the approach can be applied efficiently to those systems. The authors also provide data suggesting that their prioritized test case orders achieve coverage quickly, and can detect faults early; however, their studies do not compare their prioritized test suites to other test suites, so it is not possible to say whether the results represent an improvement in the rate of fault detection over those that would be obtained with other orderings. Kim and Porter (2002) present a technique, which they refer to as a "history-based prioritization," that utilizes information from previous testing cycles to select the test cases that must be executed for a new version of the program. This technique is not, however, a "prioritization technique" in the sense defined in the literature (and summarized earlier in this section), because it imposes no ordering on test cases-the characteristic essential to the definition of prioritization. Rather, the approach selects a subset of a test suite, using history information to determine which test cases should be selected, and is more accurately described as a "regression test selection technique" (Rothermel and Harrold, 1996) . Avritzer and Weyuker (1995) present techniques for generating test cases that apply to software that can be modeled by Markov chains, provided that operational profile data is available. Although the authors do not use the term "prioritization," their techniques generate test cases in an order that can cover a larger proportion of the software states most likely to be reached in the field earlier in testing, essentially, prioritizing the test cases in an order that increases the likelihood that faults more likely to be encountered in the field will be uncovered earlier in testing. Though not concerned with the prioritization of existing test cases for use in testing modified software, the approach provides an example of the application of prioritization in the case in which test suites are not available.. Among the papers described above, only a few (Elbaum et al., 2001b Rothermel et al., 1999 Rothermel et al., , 2001 ) report results of studies or experiments explicitly assessing the ability of prioritization techniques to improve rate of fault detection, relative to each other or to unprioritized test suites. As reported in Section 1, however, these studies show that technique effectiveness varies widely with a number of factors involving program, change characteristics, and test suite characteristics. None of the techniques investigated have proven uniformly superior to others in all situations. To make effective use of prioritization, practitioners require strategies for selecting the appropriate techniques. To date, no such strategies have been provided.

O O O B I I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I I B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Equivariant Pieri Rule for the homology of the affine Grassmannian Abstract. An explicit rule is given for the product of the degree two class with an arbitrary Schubert class in the torus-equivariant homology of the affine Grassmannian. In addition a Pieri rule (the Schubert expansion of the product of a special Schubert class with an arbitrary one) is established for the equivariant homology of the affine Grassmannians of SLn and a similar formula is conjectured for Sp2n and SO2n+1. For SLn the formula is explicit and positive. By a theorem of Peterson these compute certain products of Schubert classes in the torus-equivariant quantum cohomology of flag varieties. The SLn Pieri rule is used in our recent definition of k-double Schur functions and affine double Schur functions. Let G be a semisimple algebraic group over C with a Borel subgroup B and maximal torus T . Let Gr G = G(C((t)))/G(C [[t] ]) be the affine Grassmannian of G. The Tequivariant homology H T (Gr G ) and cohomology H T (Gr G ) are dual Hopf algebras over S = H T (pt) with Pontryagin and cup products respectively. Let W 0 af be the minimal length cosets in W af /W where W af and W are the affine and finite Weyl groups. Let {ξ w | w ∈ W 0 af } be the Schubert basis of H T (Gr G ). Define the equivariant Schubert homology structure constants d where u, v ∈ W 0 af . One interest in the polynomials d w uv is the fact that they are precisely the Schubert structure constants for the T -equivariant quantum cohomology rings QH T (G/B) [LS2, Pet] . Due to a result of Mihalcea [Mih] , they have the positivity property. Our first main result (Theorem 6) is an "equivariant homology Chevalley formula", which describes d w r 0 ,v for an arbitrary affine Grassmannian. Our second main result (Theorem 24) is an "equivariant homology Pieri formula" for G = SL n , which is a manifestly positive formula for d w σm,v where the homology classes {ξ σm | 1 ≤ m ≤ n − 1} are the special classes that generate H T (Gr SLn ). In a separate work [LS3] we use this Pieri formula to define new symmetric functions, called k-double Schur functions and affine double Schur functions, which represent the equivariant Schubert homology and cohomology classes for Gr SLn .. T.L. was supported by NSF grant DMS-0901111, and by a Sloan Fellowship. M.S. was supported by NSF DMS-0652641 and DMS-0652648.

O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O