Natural Language Processing (almost) from Scratch We propose a unified neural network architecture and learning algorithm that can be applied to various natural language processing tasks including: part-of-speech tagging, chunking, named entity recognition, and semantic role labeling. This versatility is achieved by trying to avoid task-specific engineering and therefore disregarding a lot of prior knowledge. Instead of exploiting man-made input features carefully optimized for each task, our system learns internal representations on the basis of vast amounts of mostly unlabeled training data. This work is then used as a basis for building a freely available tagging system with good performance and minimal computational requirements. Will a computer program ever be able to convert a piece of English text into a data structure that unambiguously and completely describes the meaning of the natural language text? Among numerous problems, no consensus has emerged about the form of such a data structure. Until such fundamental Artificial Intelligence problems are resolved, computer scientists must settle for reduced objectives: extracting simpler representations describing restricted aspects of the textual information.. These simpler representations are often motivated by specific applications, for instance, bag-of-words variants for information retrieval. These representations can also be motivated by our belief that they capture something more general about natural language. They can describe syntactic information (e.g. part-of-speech tagging, chunking, and parsing) or semantic information (e.g. word-sense disambiguation, semantic role labeling, named entity extraction, and anaphora resolution). Text corpora have been manually annotated with such data structures in order to compare the performance of various systems. The availability of standard benchmarks has stimulated research in Natural Language Processing (NLP) and effective systems have been designed for all these tasks. Such systems are often viewed as software components for constructing real-world NLP solutions.. The overwhelming majority of these state-of-the-art systems address a benchmark task by applying linear statistical models to ad-hoc features. In other words, the researchers themselves discover intermediate representations by engineering task-specific features. These features are often derived from the output of preexisting systems, leading to complex runtime dependencies. This approach is effective because researchers leverage a large body of linguistic knowledge. On the other hand, there is a great temptation to optimize the performance of a system for a specific benchmark. Although such performance improvements can be very useful in practice, they teach us little about the means to progress toward the broader goals of natural language understanding and the elusive goals of Artificial Intelligence.. In this contribution, we try to excel on multiple benchmarks while avoiding task-specific enginering. Instead we use a single learning system able to discover adequate internal representations. In fact we view the benchmarks as indirect measurements of the relevance of the internal representations discovered by the learning procedure, and we posit that these intermediate representations are more general than any of the benchmarks. Our desire to avoid task-specific engineered features led us to ignore a large body of linguistic knowledge. Instead we reach good performance levels in most of the tasks by transferring intermediate representations discovered on large unlabeled datasets. We call this approach "almost from scratch" to emphasize the reduced (but still important) reliance on a priori NLP knowledge.. The paper is organized as follows. Section 2 describes the benchmark tasks of interest. Section 3 describes the unified model and reports benchmark results obtained with supervised training. Section 4 leverages large unlabeled datasets (∼ 852 million words) to train the model on a language modeling task. Performance improvements are then demonstrated by transferring the unsupervised internal representations into the supervised benchmark models. Section 5 investigates multitask supervised training. Section 6 then evaluates how much further improvement can be achieved by incorporating standard NLP task-specific engineering into our systems. Drifting away from our initial goals gives us the opportunity to construct an all-purpose tagger that is simultaneously accurate, practical, and fast. We then conclude with a short discussion section. We have presented a multilayer neural network architecture that can handle a number of NLP tasks with both speed and accuracy. The design of this system was determined by our desire to avoid task-specific engineering as much as possible. Instead we rely on large unlabeled datasets and let the training algorithm discover internal representations that prove useful for all the tasks of interest. Using this strong basis, we have engineered a fast and efficient "all purpose" NLP tagger that we hope will prove useful to the community. a r X i v Collobert, Weston, Bottou, Karlen, Kavukcuoglu and Kuksa When participating in an (open) challenge, it is legitimate to increase generalization by all means. It is thus not surprising to see many top CoNLL systems using external labeled data, like additional NER classifiers for the NER architecture of Florian et al. (2003) or additional parse trees for SRL systems (Koomen et al., 2005) . Combining multiple systems or tweaking carefully features is also a common approach, like in the chunking top system (Shen and Sarkar, 2005) .. However, when comparing systems, we do not learn anything of the quality of each system if they were trained with different labeled data. For that reason, we will refer to benchmark systems, that is, top existing systems which avoid usage of external data and have been well-established in the NLP field: (Toutanova et al., 2003) for POS and (Sha and Pereira, 2003) for chunking. For NER we consider (Ando and Zhang, 2005) as they were using additional unlabeled data only. We picked (Koomen et al., 2005) for SRL, keeping in mind they use 4 additional parse trees not provided by the challenge. These benchmark systems will serve as baseline references in our experiments. We marked them in bold in Table 2 .. We note that for the four tasks we are considering in this work, it can be seen that for the more complex tasks (with corresponding lower accuracies), the best systems proposed have more engineered features relative to the best systems on the simpler tasks. That is, the POS task is one of the simplest of our four tasks, and only has relatively few engineered features, whereas SRL is the most complex, and many kinds of features have been designed for it. This clearly has implications for as yet unsolved NLP tasks requiring more sophisticated semantic understanding than the ones considered here.

B I I O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Increasing network lifetime in an energy-constrained wireless sensor network Abstract: Energy in Wireless Sensor Networks is a scarce resource, therefore an energyefficient management is required to increase the network lifetime. In this paper, we study the problem of optimal power allocation, taking into account the estimation of total signal-to-noise ratio (SNR) at the Fusion Center (FC). We consider that nodes transmit their data to the Fusion Center over quasi-static Rayleigh fading channels (QSRC). In order to analyze our approach, we will investigate first the orthogonal channels, and secondly the non-orthogonal ones introducing a virtual MISO in the communication. We consider in both cases that the nodes have Channel State Information (CSI). Simulations that have been conducted using these two channel configurations show that, thanks to our new algorithm, the network lifetime is extended by an average that can reach 82,80% compared to the network lifetime in the other methods. Wireless sensor networks (WSNs) represent a technological revolution resulting from convergence of electronic systems and wireless communication systems. A WSN is a special network composed of a large number of nodes equipped with embedded processors, sensors, and radios. These nodes collaborate to accomplish a common task such as environment monitoring or tracking by collecting data and performing local processing. Actually, there are two ways to process sensored data. In the first one, the nodes exchange information (measurements) between them, and based on this exchange each one of them gets the estimation of the measured parameter. In the second one, the data are routed over multiple nodes to reach a destination node named the Fusion Center (FC). In our work, we consider the case where all nodes have direct access to the FC. The WSNs have been widely used in several areas, such as military operations, prevention against fires, public safety. In such areas, the sensors are often randomly placed (they can be dropped from an aircraft). Due to this arbitrary distribution which is frequently done in hostile or inaccessible environments and also due to the small size of sensors, energy becomes a scarce resource in WSNs. This fact automatically influences the lifetime of the sensors and, therefore, the lifetime of the whole network. So, an efficient energy management is necessary to maximize the network lifetime. In order to solve this problem and put our work in perspective, we give a brief overview of related works. We begin by mentioning the work of (Nguyen et al. 2007 ) and (Marzetta & Hochwald 1999) which shows that considering the model of energy presented in (Cui et al. 1999) , the total power consumption of a RF system consists of two main sources, namely the transmission power P pa and the power dissipated in the circuit P c by all RF blocks. for simplicity, the blocks of digital signal processing (coding, modulation, detection...) are omitted. Based on the work in (Chouhan et al. 2009 ), in longrange applications, the energy efficient management focuses on minimizing the transmission energy only, where the circuit energy is not taken into account. However, in short-range applications the signal transmission energy required is not very high and is comparable to the circuit energy consumed in transmitting and receiving signal. Then, different approaches need to be taken into account to minimize the total energy consumption. To achieve this goal, we mention the work of Collin et al. (Collin et al. 2004) which proves that as to provide a reduced transmission power, the distance between the nodes should be reduced. Consequently, the cooperative approach is used (Nosratinia & Hedayat 2004) (Hunter & Nosratinia 2002) (Hunter & Nosratinia 2003) , where information coming from the source node passes by a number of relay nodes before reaching the destination node improving energy efficiency. Recently, Multi-antenna systems and relay transmission have been fully exploited and intensively studied in wireless networks; this is due to their potential to remarkably decrease the energy consumption. According to Belmega et al in (Belmega et al. 2010) , Multi-Input Multi-Output (MIMO) systems are more energy efficient than SISO systems if the only consumed energy that is taken into account is the one used during transmission. However, when the circuitry energy consumption is also considered, this affirmation is no longer true. Given the limited physical size of sensor nodes, they cannot carry multiple antennas at the same time, and knowing that the energy efficiency of MIMO transmission is particularly useful for WSN, a new transmission technique named "Cooperative MIMO" has been introduced in ) (Jayaweera 2006) (Abdellaoui et al. 2009 ). This technique is based on the cooperation principle where the existence of different nodes in the network is exploited to transmit the information from the source to a specific destination by virtually using the MIMO system ) (Sendonaris et al. 2003) . The Cooperative MIMO allows to obtain the space-time diversity gain (Winters 1998) , the reduction of energy consumption (Nguyen et al. 2007 ) (Jayaweera 2004) , and the enhancement of the system capacity . In our work, we consider a "Cooperative MISO" where all nodes have direct access to the FC. In (Nguyen et al. 2008), Nguyen et al. propose two techniques of cooperative reception to enhance energy efficiency. The whole space-time combination at the destination node is carried out by the first technique, while the second one independently performs signal processing and space-time combination at each cooperative node. In (Belmega et al. 2010) , Li et al. introduce the low-energy adaptive clustering hierarchy (LEACH) framework to improve the energy-efficiency where the STBC cooperative transmission (Saxena et al. 2010 ) is applied. In addition, several studies have explicitly addressed the problem of maximizing the network lifetime using various methods for minimizing the energy consumption. In (Y. Thomas Hou & Wieselthier 2007) optimal solutions are presented for maximizing a static network lifetime through a graph theoretic approach using static broadcast tree. In (Y. Thomas Hou & Wieselthier 2007) Thomas et al. have presented an optimal solution for maximizing the network lifetime through a graph theoretic approach using a static multicast tree. In (Chandrakasan 1999) (Shih 2001) , the authors have explored the fundamental limits of energy-efficient collaborative data-gathering by deriving upper bounds on the lifetime of increasingly sophisticated sensor networks. Those authors assume that sensor nodes consume energy only when they process, send or receive data. In (Winters 1998) , the authors have studied the node density vs. network lifetime trade off for a cellbased energy conservation technique. In (S. , the authors propose an optimal configuration algorithm for the case where the information passes by a single relay to reach the destination creating several boughs. Their objective is to minimize the total transmission power subject to a required error rate at the destination. In this case, the transmission power of the source and all the relays are taken to be fixed and equal. In (Y. Thomas Hou & Wieselthier 2007) , the authors have given an extension for the previous work where they have considered a strategy to enhance the performance of the previous algorithm. They have given an expression for the optimal allocation of transmission power amongst the selected relay nodes, whereby the total network power for a given system SER is minimized. In (Winters 1998) , the objective of the authors is to devise solutions that maximize network lifetime under such a minimal coverage constraint. Alfieri et al. proposed two solutions in order to maximize the network lifetime: the first one, based on column generation, must run in a centralized way, whereas the second one is based on a heuristic algorithm aiming at a distributed implementation. The main objective of the above mentioned works is to find the transmission powers when the nodes have no knowledge of their future transmissions. Thus, any optimization will have a stochastic nature. In literature, such problem is solved using the dynamic programming (Saxena et al. 2010) . The complexity of dynamic programming can be expressed as Θ(M E Q M ), where Q is the number of quantized levels of channels, M is the number of sensors, and E is the number of possible values for the residual energy of a sensor. Therefore, we are motivated to find a new approach less complex and more efficient in terms of lifetime. Referring to the mentioned previous works, MIMO system (MISO: simplified version) is important to decrease the transmission power, and knowing that the optimal power allocation depends on the measures of the quality of service (QoS), namely the outage probability, capacity, SNR, and BER. Our work consists then to virtually introduce a MISO system and to consider the SNR constraint by creating a new algorithm less complex and more efficient in terms of lifetime. We use the orthogonal channels and the non-orthogonal one taking into account the total SNR constraint at the FC. In other words, the aim of this approach is to provide the optimal transmission power in order to maximize the network lifetime while guaranteeing the required performance. This paper is organized as follows. Section II explains our method applied to the orthogonal channels and the non-orthogonal one considering that the nodes have direct access to the FC and the nodes transmit their data over a QSRC where the Channel State Information (CSI) is known by all the nodes. Section III, presents the conducted experiments and the last section concludes the paper. This paper presents a new algorithm which aims at minimizing the power consumption and consequently Figure 6 The comparison between our method and other methods concerning the network lifetime using the Non-Orthogonal channels maximize the network lifetime. This method takes into consideration the estimation of the overall SNR at the FC. Our simulations show that our new method using the orthogonal channels scheme and Non-Orthogonal one, consumes less energy than the other methods. The future work is to apply our new method to Non-Orthogonal channels assuming that the channel coefficients are unknown.

O B I O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Answering Software Evolution Questions: An Empirical Evaluation Developers often need to find answers to questions regarding the evolution of a system when working on its code base. While their information needs require data analysis pertaining to different repository types, the source code repository has a pivotal role for program comprehension tasks. However, the coarse-grained nature of the data stored by commit-based software configuration management systems often makes it challenging for a developer to search for an answer.We present Replay, an Eclipse plug-in that allows developers to explore the change history of a system by capturing the changes at a finer granularity level than commits, and by replaying the past changes chronologically inside the integrated development environment, with the source code at hand. We conducted a controlled experiment to empirically assess whether Replay outperforms a baseline (SVN client in Eclipse) on helping developers to answer common questions related to software evolution. The experiment shows that Replay leads to a decrease in completion time with respect to a set of software evolution comprehension tasks. When evolving a code base, during software development or software maintenance, developers keep a mental model of the system-an internal working representation of the software under consideration [28] . This individual understanding of the system is constantly being updated by the developer's interactions with the code and the team, and by seeking answers to various questions [26, 2, 6, 15] . These questions span multiple areas [25] such as program comprehension, software evolution, collaborative software development, and program analysis; therefore, they require a variety of information sources (e.g., colleagues, code bases, issue trackers, documentation, communication history), and multiple tools (e.g., [27, 17, 1, 30] ) to fulfill them.. Although there are a number of resources (data and tools) available to ease the comprehension of a system and its evolution, the amount of resources actually used by developers is often limited to talking to colleagues and exploring the code.. In an exploratory study [19] , LaToza et al. report that: 1) almost all teams have a team historian, who is the go-to person for questions about the code; 2) most team members subscribe to the check-in messages to keep themselves updated with the code evolution, though many of them expressed dissatisfaction with the lack of detail provided by their teammates when describing the changes in commit messages.. We argue that this lack of detail is a fundamental problem for understanding software evolution, i.e., changes made by other developers. The problem is related to the coarse granularity at which changes are checked in and, consequently, seen by others. When trying to understand the evolution of the code, the delta between subsequent changes can be complex enough to prevent developers from inferring the design decision behind the changes in the code. Moreover, as indicated by previous studies [8, 3] , large commits can also lead to merge conflicts, duplicated work, and conflicting design decisions .. In our previous work [10, 12] we presented Syde, an Eclipse plug-in that records fine-grained changes in multi-developers projects by continuously tracking code edits performed in the Integrated Development Environment (IDE).. In recent work we presented Replay [13, 11] , an Eclipse plug-in that allows developers to explore the rich change repository created by Syde. Developers can search for fine-grained changes made by a set of people to a set of artifacts and watch them in the chronological order as originally performed in the IDE. This counts for a better user experience [21] than the aggregated form of commit-based Software Configuration Management (SCM) tools, such as CVS and Subversion.. In a previous version of this paper [11] , we conducted a controlled experiment to assess whether Replay is at least as effective and efficient as the state of the practice at supporting developers with their questions related to software evolution [11] . The design of the experiment involved the selection, from previous catalogues [26, 2, 6, 15] , of a set of common questions that developers ask. We converted them into a set of tasks to measure both the correctness of the task solutions and their completion time. In this extended version of the paper, we conducted additional runs of the controlled experiment, involving new participants, we expand our analysis of the experiments results, and we make our experiments replicable by sharing the necessary information.. The contributions of this article are:. 1. Replay, a toolset to replay and exploit a fine-grained change software repository, thus aiding developers in answering their questions related to software evolution;. 2. a report on the design and operation of a series of controlled experiments to compare the performance of Replay with the baseline tool (SVN client) in performing selected software evolution comprehension tasks;. 3. an analysis of the results, which shows a statistically significant advantage of Replay over the baseline in time, and indicates advantages of Replay on correctness;. 4. the complete experimental data to make our experiment replicable.. Structure of the article. In Section 2 we review Syde and its change model to subsequently present Replay. In Section 3 we describe the design and operation of our controlled experiment. In Section 4 we analyze the experiment results and discuss the by Syde. Since atomic changes are too fine-grained to be shown individually, Replay groups them by timestamp, author and artifact (package or class), i.e., all the changes that were performed by a developer in a class between two subsequent builds are grouped together based on the last build's timestamp. Within a group there cannot be more than one change to one artifact, thus we maintain the granularity of the changes.. The aim of this experiment is to compare tool efficiency in supporting software practitioners understanding the change history of a software system.. You will use Eclipse with the Replay plug-in to analyze Spreadsheet, a spreadsheet application written in Java by undergraduate students at the University of Lugano.. You are going to Perform 6 tasks, with limited time to solve 5 of them. You have 10 minutes to solve each of the first 4 tasks, and 20 minutes to solve task 5.. We kindly ask you:. • to write down your answers in a legible way;. • not to consult any other participant during the experiment;. • to perform the tasks in the specified order;. • to write down the current time before starting to work on a new task (after reading it) and once after completing all the tasks; • to announce to the experimenter that you are starting to work on a new task (after reading it), in order to reset your allocated timer; • not to return to earlier tasks because it affects the timing; • to fill in the required information for each task. In the case of multiple choices check the most appropriate answer and provide additional information, if requested.. In the following, there is one warm-up task for you to get used to the tool and the experiment.. The experiment is concluded with a short debriefing questionnaire. We presented Replay, a tool that allows developers to explore the evolution history of a system by chronologically replaying the fine-grained changes collected by Syde. We argue that Replay can be useful to help developers in finding answers to questions they raise during development and maintenance that are related to the evolution of a system.. We conducted a controlled experiment to evaluate whether Replay is at least as effective and efficient as the state of the practice at supporting developers with their questions related to software evolution. The results indicate that Replay leads to an improvement in both correctness (16.76%) and completion time (11.56%), with the latter being statistically significant at 99% confidence interval. As an indication of a superior performance of the experimental group in terms of correctness, 75% of this group performed better than or equal to 50% of the control group. In terms of completion time, 50% of the experimental group was faster than (or equivalent to) 25% of the control group. Mir 50% faster than 25%?? These results show that there are benefits in using Replay over the state of the practice tools for most of the tasks included in this empirical evaluation.. The per-task analysis of the results provided a number of insights on the type of tasks our approach supports best. For tasks that needed fine-grained change information, or in which the the inspection of recent changes was important, Replay outperformed the baseline. However, when the tasks required a high-level overview of the changes, Replay did not perform better than the baseline.. As future work, we plan to incorporate the suggestions given by the subjects to improve the usability of the tool. Some planned enhancements are to: offer the possibility to aggregate the changes when a general overview is needed; implement search bars in the Replay view and in the filters; provide a history of recent requests; and add a navigation mode to the Replay view that allows restricting the navigation to changes of a single developer. Approaches Related to Replay To our knowledge, Replay is the first tool to support replaying development sessions in a collaborative environment. However, other tools support programmers with their quests. Fritz and Murphy propose a prototype that combines different information fragments (source code, work items, change sets, teams, comments, wiki) to support 78 questions software developers ask about a development project [6] . The tool Ferret combines four different sources of information (static, dynamic, evolutionary, and Eclipse PDE) to build a knowledge base for answering conceptual queries [2] . James is a knowledge base, composed of IDE interactions and micro-blogging, to support developers with their quests [9] .. Looking at our work in the broader context of software evolution, there are various lines of research that relate to ours. In the software evolution analysis context, several approaches make use of the changes performed to a system over its lifetime to support its comprehension: Lanza, Gîrba et al., and Lungu et al. summarize and visualize respectively the evolution of classes [16] , the evolution of class hierarchies [7] , and the evolution of inter-module relationships [20] . In these works the history is not replayed, but summarized; and the order in which the changes are performed is lost. We specifically focus on replaying the change events in the order in which they happened.. A few approaches focus on replaying the changes that happened in a system. Wettel and Lanza visualize the evolution of the entire system by allowing the user to travel in time and observe the changes of the system as they are represented in a 3D city metaphor [29] . Hindle et al. present an animation of the evolution of the architecture of a system [14] in the Yarn tool. The animation presents the evolution of the relationships between the modules of the system. These approaches allow the animation of the changes, but present the changes at a high level of abstraction, from which the code is not accessible.. One major difference between our work and the ones aforementioned is the level of detail of the data. In most of the approaches the data is extracted from commit-based SCM systems, implying that changes between versions can be arbitrarily complex. An approach that uses fine-grained change information was proposed by Robbes [23] . Although he collects fine-grained information from software systems, Robbes does not use it to support the replaying of the changes, but he exploits it for other purposes, e.g., to detect and characterize development sessions [25] . Dig, instead, uses a change-centric approach to record sequences of refactorings, and to replay them on other library-based applications [4] .. Empirical Studies There are relatively few empirical studies by means of controlled experiments in software engineering. Further, there is no controlled experiment that directly relates to ours: answering developers' questions related to software evolution. However, there are a number of controlled experiments related to software evolution and program comprehension.. Quante evaluates, through a controlled experiment, the support provided by dynamic object graphs on answering a set of program comprehension tasks [22] . Cornelissen et al. performed a controlled experiment to evaluate the use of Extravis, an execution trace visualization tool, to answer program comprehension tasks [1] . Wettel et al. assess the use of CodeCity to perform program comprehension and quality assessment tasks [30] .. The major difference between these controlled experiments and ours is that they evaluate tools that visualize data other than source code (dynamic graphs, execution traces, system models) to support program comprehension. We evaluate a tool that allows a developer to investigate the history of the system by looking directly at its source code.

O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


High-performance Java codes for computational fluid dynamics The computational science community is reluctant to write largescale computationally-intensive applications in Java due to concerns over Java's poor performance, despite the claimed software engineering advantages of its object-oriented features. Naive Java implementations of numerical algorithms can perform poorly compared to corresponding Fortran or C implementations. To achieve high performance, Java applications must be designed with good performance as a primary goal. This paper presents the objectoriented design and implementation of two real-world applications from the field of Computational Fluid Dynamics (CFD): a finitevolume fluid flow solver (LAURA, from NASA Langley Research Center), and an unstructured mesh adaptation algorithm (2D_TAG, from NASA Ames Research Center). This work builds on our previous experience with the design of high-performance numerical libraries in Java. We examine the performance of the applications using the currently available Java infrastructure and show that the Java version of the flow solver LAURA performs almost within a factor of 2 of the original procedural version. Our Java version of the mesh adaptation algorithm 2D_TAG performs within a factor of 1.5 of its original procedural version on certain platforms. Our results demonstrate that object-oriented software design principles are not necessarily inimical to high performance. The Java programming language has many features that are attractive for both general-purpose and scientific computing. Among them are: support for object-oriented concepts such as inheritance, encapsulation, and polymorphism that allow the abstraction of common physical concepts and the development of reusable class libraries for them; the architecture-neutrality of Java byte code, which enables portability across multiple platforms; garbage collection, which simplifies memory management; and language-level support for multithreading, which allows parallel applications to be developed more easily in Java. However, scientific programs written *This work was supported in part by NSF Grant CCR-9711438.. Permission to make digital or hard copies ot'all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commcrcial advantage and that copies bear this notice and the lull citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. JAVA Grandc/ISCOPE 01 Palo Alto ('A USA Copyright ACM 2001 1-58113-359-6/01/06...$5.00 in Java usually run slower than corresponding programs written in Fortran and C, due either to the intrinsic overhead of these features or to the relative immaturity of current Java Virtual Machine (JVM) implementations. Because high performance is a primary concern in computational science, that community has been reluctant to adopt Java as the language of choice for numerical applications.. To demonstrate the viability of high-performance computing in Java and to encourage its greater adoption in the computational science community, several authors have ported numerical libraries to Java [1, 5, 19] , written Fortran-to-Java translators [8, 10] , developed compilation technology for Java [6, 7, 25] , and written class libraries to address deficiencies in the Java language for numerical computing [26] . Although these studies demonstrate the potential of Java for high-performance computing, several factors limit their usefulness in determining whether large-scale scientific applications written in Java can achieve high performance. First, numerical and class libraries form only part of such applications. The only previous ports of large-scale codes to Java that we know of are a geophysical simulation by Jacob et at. [20] and a parallel multi-pass renderer by Yamauchi et al. [31] . Second, "line-by-line" translations of procedural codes written in Fortran or C to Java do not exploit object-oriented techniques, one of the main advantages to programming in Java.. The primary goal of the work described in this paper is to demonstrate that realistic scientific applications can be written in Java that make full use of the language's object-o•nted capabilities and still show good performance on current standards-conforming JVM implementations. Secondary goals include characterizing the performance and identifying the bottlenecks in different JVM implementations, identifying design principles for portable high-performance object-oriented software in Java, and making available additional benchmarks for the high-performance Java community. It is our thesis that in order to achieve high performance with such applications, one must consciously design for performance in both the definition and the implementation of the software components that comprise them. We choose two real-world examples from the field of Computational Fluid Dynamics (CFD) for our study. The first example is the Langley Aerothermodynamic Upwind Relaxation Algorithm (LAURA) [9, 12] , a finite-volume flow solver developed at NASA Langley Research Center for multiblock, structured grids. LAURA has been widely used to compute hypersonic, viscous, reacting-gas flows over reentry vehicles such as the Shuttle Orbiter [14] , the Mars Pathfinder [24] , and the X-33 Reusable Launch Vehicle [15] . The second example is the Two-Dimensional Triangular Adaptive Grid (2D_TAG) code [27] developed at NASA Ames Research Center for adaptation of unstructured meshes. This !code is a simplified version of a three-dimensional mesh adaption :algorithm [4] developed primarily for helicopter flowfield simulations. Both LAURA and 2D_TAG are currently used in government ;and industry and are well-documented.. The remainder of the paper is organized as follows. Sections 2 land 3 describe the algorithms and data structures of the original versions of LAURA and 2D_TAG and their object-oriented Java implementations. Section 4 presents performance measurements of the two codes on several platforms. Section 5 concludes and i presents plans for future work. We have described the design, implementation, and evaluation of object-oriented "100% Pure" Java codes for two representative CFD applications that differ radically in their characteristics. The flow solver LAURA is structured, largely static, and floating-point intensive. The mesh adaptation algorithm 2D_TAG is unstructured, very dynamic, fine-grained, and limited by the speed of object manipulation. We have demonstrated that LAURA's running time is almost within a factor of 2 and 2D_TAG's is within a factor of 1.5 of their optimized native, procedural counterparts using current offthe-shelf Java compilers and JVM technology. We feel that this level of performance is extremely promising and is susceptible to further improvement as Java technology matures.. Performance should of course be considered in the larger context of the software design cycle. In both codes, the Java versions are smaller in size than the original versions: 5300 lines of Java to 14500 lines of Fortran for LAURA, and 1300 lines of Java to 1900 lines of C for 2D_TAG. In addition, the Java versions are much more modular and hierarchically structured. The Java version of LAURA contains 49 classes organized in a three-deep inheritance hierarchy, while the O-O version of 2D_TAG contains eight classes in a two-deep hierarchy. Such modularization makes it easier to extend the functionality of the code, e.g., to add different boundary conditions, gas chemistry options, or relaxation algorithms in LAURA. Programmer productivity also needs to be considered. The Java versions of both codes were designed and implemented in 18 months by a single programmer with minimal Java experience at the beginning of the effort. The original versions, by contrast, represent many person-years of effort.. The performance of the codes indicates the complex design tradeoffs in JVM implementations and highlights several aspects of JVM behavior that should be investigated further for high-performance Java. First, the user can control garbage collection activity only in very indirect ways, such as by specifying the initial and maximum heap sizes. The wide variety of GC algorithms, their different performance characteristics, and the black-box nature of this component of the JVM, make this level of user interaction inadequate for high-performance applications. Greater user interaction with the garbage collector within the constraints of the Java platform need to be investigated. Second, the performance of multithreading varies widely among platforms, and needs to be improved to scale to large numbers of threads.. Regarding future work, we plan to implement a parallel, threedimensional version of the 2D_TAG code as well as continue our experimentation. ; LAURA solves the system of partial differential equations gnv-: eming fluid flow numerically by first discretizing the solution domain using a collection of blocks, points, and cells as shown in Figure 1 . A block is defined as a rectangular collection of cells. Figure 1 shows a single block. Because the grid is structured, a multidimensional array is the natural data structure to use for storage. Solution variable unknowns are assumed to lie at the center of a cell. Numerically approximating the physically relevant fluxes ! and stresses acting on the cell walls leads to a finite-volume formu-' lation of the governing conservation laws. In LAURA, the upwindbiased inviscid flux is constructed using Roe's flux-difference splitting [29] and Harten's entropy fix [16] with second-order corrections based on Yee's symmetric total-variation-diminishing (TVD) 'scheme [32] .. To drive the residual of the finite-volume approximation scheme to zero simultaneously at all the cells of the domain, LAURA uses a point-implicit relaxation strategy obtained by treating the variables at the local cell center L at the advanced iteration level and using the latest available data from neighboring cells. This governing relaxation equation is. where ML is the n x n point-implicit Jacobian, qL is the vector of conserved variables, rL is the the residual vector, and n is the number of unknown variables. The value of n varies from 5 for perfect gas flows to 15 for chemically-reacting flows with eleven species. The point-implicit strategy has several implications compared with other solvers (e.g., tridiagonal, conjugate-gradient). First, the memory requirements are lower, which allows solutions using nonequilibrium chemistry to be obtained over larger grids. This savings in memory is traded off against an increase in the number of iterations (albeit faster iterations) required to reach convergence. Second, iterations across blocks do not have to be synchronized. This allows for an efficient parallel implementation.. The algorithm may be summarized as follows:. 1. Sum the fluxes for each cell in the domain to get the residual 'rL.. 2. Compute the change in conserved variables at each cell, t~qL, by solving equation (1) using Gaussian elimination.. 3. Update the vector of conserved variables qL at each cell.. These steps constitute an iteration in which the solution variables at all cells in the domain are integrated to the next time level. While the above description captures the basic steps in the algorithm, the original Fortran version of LAURA contains additional gather and scatter steps at each iteration designed to take advantage of vector architectures. At each iteration, two-dimensional slabs of size IMAX x JMAX are copied into one-dimensional temporary arrays, the fluxes and variables of each slab are updated, and then the temporary values are put back into its original two-dimensional slab. This approach works very well on vector architectures as it creates large vectors on which the vector processors can operate. However, the extraneous copying hurts performance on current, RISC-based architectures that use caches. To make a more accurate measure of the relative performance of the Java version to the original procedural version, we wrote a corresponding C version of LAURA that eliminates the unnecessary copying.. A mesh is described as a collection of two-dimensional triangular elements, as shown in this strategy is that relatively few mesh points need to be added or deleted at each refinement/coarsening step. A disadvantage is that : complicated data structures and logic are required to keep track of the mesh objects (e.g., vertices, edges, elements) and their neighbors as the objects are added and removed. The algorithm involves , much "pointer chasing" leading to irregular and dynamic data acicess patterns.. Triangular elements may be refined by bisecting one or more of its edges. An isotropic subdivision bisects all three of an element's edges to form four congruent "child" triangles. An anisotropic subdivision bisects one or two of the edges to form two or three child triangles. Figure 5 depicts this process of triangle refine-,ment. Anisotropic subdivisions are required as a transition from i isotropically-refined elements to unrefined elements. Coarsening is the process of deleting child edges and elements and restoring their parents.. The elements and edges of the mesh are stored as forests, with a tree for subsets of elements and edges. Using trees simplifies i the coarsening process. Roots of trees in the forest correspond to the original unrefined elements and edges. Each level of a tree I in the forest corresponds to a level of refinement. Leaves of the trees represent elements and edges at the maximum level of refinement. Leaf elements and edges are the only ones that may be refined or coarsened. Vertices do not have parents or children and so are stored in an array or list. The refinement algorithm may be summarized as follows:. I. For each leaf edge of the mesh that meets a specified error criterion, mark edge for refinement.. 2. Subdivide elements based on their edges that are marked for refinement.. 3. Add new vertices, edges, and elements to the mesh.. The coarsening algorithm is similar. As described, this mesh adaption algorithm involves multiple passes over the lists of edges and elements.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Smallest independent dominating sets in Kronecker products of cycles Let k¿2; n=2 k +1; and let m0; : : : ; m k−1 each be a multiple of n. The graph Cm 0 ×· · ·×Cm k−1 consists of isomorphic connected components, each of which is (n − 1)-regular and admits of a vertex partition into n smallest independent dominating sets. Accordingly, (independent) domination number of each connected component of this graph is equal to (1=n)th of the number of vertices in it. ? By a graph is meant a ÿnite, simple and undirected graph. The Kronecker product G × H of graphs G = (V; E) and H = (W; F) is deÿned as follows: V (G × H ) = V × W and E(G × H ) = {{(u; x); (v; y)}: {u; v} ∈ E and {x; y} ∈ F}. This product (that is variously known as direct product, cardinal product, categorical product, tensor product and cross product) is one of the most important graph products, with applications in a number of areas. It is commutative and associative in a natural way. Let C n denote the cycle on vertices 0; : : : ; n − 1; where adjacencies are deÿned in the natural way.. Let S be a vertex subset of a given graph G = (V; E). S is said to be a dominating set of G if every x ∈ V is either an element of S or is adjacent to at least one element of S. A dominating set whose elements are mutually nonadjacent in G is called an independent dominating set of G; and an independent dominating set of least cardinality is called a smallest independent dominating set (s.i.d.s.). Further, if S is such that every x ∈ V is either in S or is adjacent to exactly one element of S; then S is called a perfect dominating set of G. By (independent) domination number of G is meant the cardinality of a smallest (independent) dominating set of G.. The general problem of obtaining a smallest (independent) dominating set is NP-hard even for bipartite graphs [2] . In fact, an s.i.d.s. is not even approximable in polynomial time within a factor of n 1− for any ¿ 0 unless P = NP [5] . Domination in graphs has a number of applications in areas such as game theory, coding theory, channel assignment and resource placement. Accordingly, it has an extensive literature, cf. Haynes et al. [6] . Perfect dominating sets, in particular, are directly relevant to error-correcting codes. They have been studied in various contexts, Hamming codes being the most important [8, 14] . Perfect codes with respect to Cartesian-product graphs have been treated by Kratochvil [11] , and Livingston and Stout [12] . Additional references on domination in this product include KlavÄ z ar and Seifter [9] and Gravier and Mollard [4] . Nowakowski and Rall [13] present a systematic approach to graph invariants (including domination number) on graph products. Domination in Kronecker-product graphs has been studied by several authors [1, 3, 10] . The present paper presents a vertex partition of Kronecker products of certain cycles into smallest (independent) dominating sets. In fact, each set in the partition is a perfect dominating set.

B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O B O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O


On the relationships between models in protocol verification We formally investigate the relationships between several models that are widely used in protocol verification, namely variants of the inductive model of message traces inspired by Paulson's approach, and models based on rewriting. More precisely, we prove several over-approximation relationships between models, i.e. that one model allows strictly more traces or reachable states than the other. This is common in verification: often an over-approximation is easier to prove correct than the original model, and proving that the over-approximation is safe implies that the original model is safe-provided that the models are indeed in an over-approximation relation. We then show that some over-approximations are not sound with respect to a common formalization of authentication goals based on exchanged messages. The precise formal account that we give on the relation of the models allows us to correct the situation. Security protocols have been intensively studied using formal methods in the past 25 years, and a large variety of formalisms, models, and techniques have been developed in order to cope with the different kinds of infinity that arise in verification. Most notably, there is no bound on the number of sessions (protocol runs) that can be executed in parallel, and there are infinitely many messages an intruder can construct from what he has observed on the network, even assuming that he cannot break the cryptography. The variety of different models results in a number of problems: it is hard to tell whether the statements proved by different verification tools are equivalent, how tools can be combined, and whether a particular meta-argumentation about one model (e.g. that certain restrictions are without loss of generality) can be carried over to other models.. In this paper, we consider the relationships between variants of two widely used models. The first model is based on set rewriting, which is for instance employed as the input language for the verification tools that are part of the AVISPA-tool [2] , the CAPSL framework [16] and [22] . The second model is inspired by Paulson's approach of message traces based on the Isabelle theorem prover [24] . This latter model has influenced a number of automated verification approaches as discussed below.. Although we consider two particular models, the ideas presented in this paper are general. There are several works that argue for the equivalence of models in different formalisms [12] and the universality of their results [14] . However, it is not the focus of this paper to discuss all design choices and to compare with all existing formalisms. Rather we want to point out that most current protocol models follow the same basic idea as our set rewriting model: the protocol is a set of processes (the honest agents) each of which has a local state that is updated when receiving a message and sending an answer, and these processes are connected via an intruder-controlled network. And we thus see the set rewriting approach as a representative of the "standard" way to model protocols. In contrast, the message trace model has no explicit notion of processes or local state; rather we have rules that tell us how a given trace of exchanged messages can be extended with further messages.. The focus of this work is on the over-approximation relationship between two models, i.e. when one model allows strictly more traces or reachable states than the other. Over-approximations can induce attacks that are of type false positive, i.e. attacks that work only in the over-approximated model, but not in the original model. The false positives are thus in some sense artifacts created by the over-approximation. For falsification (i.e. detecting attacks) this is problematic, as the "real" attacks may be buried under false positives. On the other hand, for verification (i.e. trying to prove a protocol correct) over-approximation does make sense: given a precise model and an over-approximation of it, proving that the over-approximation is safe is often much easier than in the original model and immediately implies that the precise model is safe as well.. Our contributions are as follows (see also Fig. 1 ): We formally prove that the message trace model T is an over-approximation of the set rewriting model R, i.e. it contains many traces that have no counter part in the set rewriting model (or in reality), while conversely all reachable states of the set rewriting model have their counter part in the message trace model. Moreover, we show that a certain variant P of the set rewriting model is equivalent to the message trace model in a sense that is made precise in this paper, yielding a precise understanding of the relationship between the two models. This relation has never been expressed in the literature before.. We further show how the over-approximations inherent in R and T can be used to abstract away a large part of the control structure of protocols, and this control abstraction is the basis of many automated verification approaches [7, 9, 10, 11, 13, 18, 25] . We formally prove that these models (F and E) are over-approximations of the message trace model, a fact which has also never been formally analyzed before.. These relationships yield a better understanding of these models. While it is immediate that the over-approximations of the different models are sound for reasoning about secrecy, it turns out that it is not sound in general for authentication goals, i.e. a protocol may have an authentication flaw in the set rewriting model, but appear flawless in the over-approximation. We attribute the discovery of this subtlety to the rigid formal study of the models and their relationship. We also show how to soundly encode authentication goals (based on data abstraction) in the models F and E.. The work closest related to ours is [8] which considers two models similar to P and E and shows that every attack against secrecy in the original model can be recast in the over-approximation. It does not examine, however, all the relationships between models as we do, and does not consider authentication goals. Also, several papers like [23] deal with the safe over-approximation by data abstraction, which is however orthogonal to this paper. Fig. 1 gives an overview of the protocol models we consider and the theorems about their relationship. We proceed as follows: after summarizing preliminaries in Sec. 2, we define the set rewriting model and the message trace model in Sec. 3. In Sec. 4 we formally show that the message trace model over-approximates the set rewriting model. In Sec. 5 we show how to use the over-approximation to perform a complete control abstraction. In Sec. 6 we consider the specification of security goals for the protocols in the different models, based on our results about their relationship. In Sec. 7 we conclude with a summary and an outlook on future work. Observe that this work does not report on any experimental results. The reason is that this paper is only concerned with the questions of modeling protocols and comparing such models, while it is not concerned with the particular techniques that are used to automatically verify protocols based on these models. However, there exists a wide variety of empirically succesful analysis techniques based on the models presented in this paper [7, 9, 10, 11, 13, 18, 25, 24, 2] .. We have formally proved equivalence and over-approximation relationships between several protocol models. As a representative of standard protocol models, we have used the set rewriting model R. We showed that it is an over-approximation of the message trace model T in a certain sense. For a more precise account we have defined a persistent variant P of the set rewriting model; showing that this variant lies strictly between R and T demonstrates that there are in fact two orthogonal kinds of over-approximation. We then considered a farther going kind of control abstraction, basically turning from reachable states (traces) to reachable facts F (events E). We show that these models, which are often used in the context of data abstraction, are themselves over-approximations of P and T.. We then turned to the question how to define security goals in these models. For secrecy goals, it is straight-forward that the over-approximation is appropriate in the sense that any attack against secrecy in the R can be recast in the P, T, F, and E models. Thus, when we verify in one of these over-approximated models that a protocol achieves its secrecy goal, then this also holds in the original R model. It turns out that the over-approximation is not appropriate in this sense for checking authentication goals: we gave an example of a protocol and authentication goal that is violated according to the R model, while the same protocol and goal appear flawless in the T model. However, when using auxiliary events to formulate authentication, as it is done in a number of recent approaches, the over-approximation indeed works. More precisely, we can show that verification of authentication goals in T model implies there verification in the R model. Finally, we have shown how to formulate authentication for F and E using the data abstraction.. We see the main contribution of this paper by giving a precise account of several widely used models, and from that the ability to recognize such subtle problems as the one about authentication goals, and to give a provenly correct reformulation.. Besides a better understanding of the employed models, these results also pave the way for combining methods based on these models, in particular, connecting automated verification procedures with a formalization in the theorem prover Isabelle in the style of [24] . Such a connection is motivated by the fact that a protocol verification tool is not unlikely to eventually verify a flawed protocol because of a bug in that tool, while the chance that a wrong security proof is accepted by Isabelle is much smaller. (Left aside mistakes and abstractions in the formalization itself.) The results of this work give an idea how such a connection may be possible, namely that the proof object generated by an automated verification tool would be an invariant on the set of messages that can ever be sent by an honest agent, and what messages the intruder can derive. We plan to investigate the details of such a connection as future work.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Time-Bounded Reachability in Tree-Structured QBDs by Abstraction Abstract-This paper studies quantitative model checking of infinite tree-like (continuous-time) Markov chains. These treestructured quasi-birth death processes are equivalent to probabilistic pushdown automata and recursive Markov chains and are widely used in the field of performance evaluation. We determine time-bounded reachability probabilities in these processeswhich with direct methods, i.e., uniformization, results in an exponential blow-up-by applying abstraction. We contrast abstraction based on Markov decision processes (MDPs) and interval-based abstraction; study various schemes to partition the state space, and empirically show their influence on the accuracy of the obtained reachability probabilities. Results show that gridlike schemes, in contrast to chain-and tree-like ones, yield extremely precise approximations for rather coarse abstractions. In this paper we have shown that abstraction techniques allow us to quickly compute highly-accurate time-bounded reachability probabilities for a class of infinite-state CTMCs, namely tree-structured QBDs. In queueing theory this model class is widely used, however, mostly for steady-state analysis.. Computing time-bounded reachability probabilities for treestructured QBDs has not been possible before. We present non-trivial partitionings for the infinite state-space of treestructured QBDs that preserve different properties and discuss how they formally relate. In six experiments we thoroughly compare the efficiency of these partitionings, regarding the size of the resulting state space and the difference between upper and lower probability bounds, resulting from abstraction. Grid-like schemes that neglect the order of the jobs in the queue, have shown to perform best when the empty state is chosen as goal state. When the cut level is chosen according to the utilization and the choice of the initial and the goal state, we achieve differences between the upper and the lower probability bound of only 10 −6 . To reach the empty state, all jobs need to be processed, regardless of their ordering, hence grid abstraction contains all necessary information and hence produces very tight bounds. In case goal states belong to higher levels k > 0, we present a refinement of the grid abstraction, where the state-space up to level k is not abstracted and grid abstraction is used for all higher levels. This produce extremely tight bounds for this setting as well.. Even though the tree-structured QBDs in this paper result from M|PH|1 queues, our approach can also be used for the more general PH|PH|1 queues. Future work will include the automation of the parameterization, as well as finding partitionings for other highly-structured, fast growing state spaces, such as for multi-class queueing networks.

O B O O O O B O O O O O O O O O O B I O O O O O O O O O O O O O B I O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O B I O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Feedback-Based Closed-Loop Carrier Synchronization: A Sharp Asymptotic Bound, an Asymptotically Optimal Approach, Simulations, and Experiments Abstract-We derive an asymptotically sharp bound on the synchronization speed of a randomized black box optimization technique for closed-loop feedback-based distributed adaptive beamforming in wireless sensor networks. We also show that the feedback function that guides this synchronization process is strong multimodal. Given this knowledge that no local optimum exists, we consider an approach to locally compute the phase offset of each individual carrier signal. With this design objective, an asymptotically optimal algorithm is derived. Additionally, we discuss the concept to reduce the optimization time and energy consumption by hierarchically clustering the network into subsets of nodes that achieve beamforming successively over all clusters. For the approaches discussed, we demonstrate their practical feasibility in simulations and experiments. I N recent years, sensor nodes of extreme tiny size have been envisioned [1] , [2] , [3] . In [4] , for example, applications for square-millimetre-sized nodes that seamlessly integrate into an environment are detailed. At these small form factors, transmission power of wireless nodes is restricted to several microwatts. Communication between a single node and a remote receiver is then only feasible at short distances. It is possible, however, to increase the maximum transmission range by cooperatively transmitting information from distinct nodes of a network [5] , [6] . Cooperation can increase the capacity and robustness of a network of transmitters [7] , [8] and decrease the average energy consumption per node [9] , [10] , [11] .. Related research branches are cooperative transmission [12] , collaborative transmission [13] , [14] , distributed adaptive beamforming [15] , [16] , [17] , [18] , collaborative beamforming [19] , or cooperative/virtual MIMO for wireless sensor networks [20] , [21] , [22] , [23] . One approach is to utilize neighboring nodes as relays [24] , [25] , [26] as proposed by Cover and El Gamal [27] . Cooperative transmission is then achieved by multihop [28] , [29] , [30] or data flooding [31] , [32] , [33] , [34] approaches. The general idea of multihop relaying based on the physical channel is to retransmit received messages by a relay node so that the destination will receive not only the message from the source node, but also from the relay. In data flooding approaches, a node will retransmit a received message at its reception. It has been shown that the approach outperforms noncooperative multihop schemes significantly. In particular, the transmission time is reduced as compared to traditional transmission protocols [35] .. In these approaches, nodes are not tightly synchronized and transmission may be asynchronous. Synchronous transmission, however, is achieved by virtual MIMO techniques. In these implementations, identical RF carrier signal components from various transmitters that function as a distributed beamformer are superimposed. When the relative phase offset of these carrier signal components at a remote receiver is small, the signal strength of the received sum signal is improved. In virtual MIMO for wireless sensor networks, single antenna nodes are cooperating to establish a multiple antenna wireless sensor network [21] , [20] , [22] . Virtual MIMO has capabilities to adjust to different frequencies and is highly energy efficient [23] , [11] . However, the implementation of MIMO capabilities in WSNs requires accurate time synchronization, complex transceiver circuits, and signal processing that might surpass the power consumption and processing capabilities of simple sensor nodes.. Other solutions proposed are open-loop synchronization methods such as round-trip synchronization [36] , [37] , [38] . In this scheme, the destination transmits beacons in opposed directions along a multihop circle in which each of the nodes appends its part of the overall message to the beacons. Beamforming is achieved when the processing time along the multihop chain is identical in both directions. This approach, however, does not scale with the size of a network.. Closed-loop feedback-based approaches include fullfeedback techniques, in which carrier synchronization is achieved in a master-slave manner. The phase offset among the carrier signals of destination nodes is corrected by a receiver node. Diversity between RF-transmit signal components is achieved over CDMA channels [39] . This approach is applicable only to small network sizes and requires sophisticated processing capabilities at the source nodes. A more simple and less resource demanding implementation is the 1-bit feedback-based closed-loop synchronization considered in [39] and [40] . The authors describe an iterative process in which n source nodes i 2½1; . . . ; n randomly adapt the phases i of their carrier signal <ðmðtÞe jð2ðfcþfiÞtþiÞ Þ. Here, mðtÞ denotes the transmit message and f i denotes the frequency offset of node i to a common carrier frequency f c . Initially, i.i.d. phase offsets i of carrier signals are assumed. When a receiver requests a transmission from the network, carrier phases are synchronized in an iterative process. We have considered randomized search approaches to solve the problem of distributed adaptive transmit beamforming. In an analytic consideration, an asymptotically tight bound on the expected optimization time of Âðn Á k Á logðnÞÞ was derived.. Additionally, a protocol to further reduce the optimization time and energy consumption of distributed adaptive beamforming was introduced. In this protocol, the problem was divided into subproblems that were solved iteratively. Since the decrease in the synchronization time is greater Fig. 8 . RF signal strength and relative phase shift of received signal components for a network size of 100 nodes after 10,000 iterations. Nodes are distributed uniformly at random on a 30 m Â 30 m square area and transmit at P T X ¼ 1 mW. Fig. 9 . Distributed adaptive beamforming with a network size of 100 nodes where phase alterations are drawn uniformly at random. Each node adapts its carrier phase offset with probability 0.01 in one iteration. In this case, multivariable equation are solved to determine the optimum phase offset of the carrier signal. Furthermore, an asymptotically optimal algorithm was derived. For this approach, we considered the possibility to estimate the unknown feedback function by an individual node so that an optimization approach is possible that scales linearly with the network size n. This approach is asymptotically optimal since each carrier signal has to be considered at least once individually in order to find its optimum phase offset.. In mathematical simulations, we demonstrated the effect of several configurations for distributed adaptive transmit beamforming with uniform and normal distributed phase alteration methods. Generally, a low mutation probability translates to a better performance in the phase synchronization process. An adaptive probability over the course of the optimization might further improve the optimization speed. While a moderate mutation probability is beneficial at the beginning of the simulation, a smaller mutation probability shows an improved optimization speed later in the process. Also, our implementation of the asymptotically optimal method greatly outperforms the global random search approach in the synchronization achieved and the optimization speed.. Finally, in an instrumentation with USRP software radios, we demonstrated the feasibility of distributed adaptive transmit beamforming in a concrete implementation with up to four transmitters. . For more information on this or any other computing topic, please visit our Digital Library at www.computer.org/publications/dlib.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O