Modularity of Convergence and Strong Convergence in Infinitary Rewriting Abstract. Properties of Term Rewriting Systems are called modular iff they are preserved under (and reflected by) disjoint union, i.e. when combining two Term Rewriting Systems with disjoint signatures. Convergence is the property of Infinitary Term Rewriting Systems that all reduction sequences converge to a limit. Strong Convergence requires in addition that redex positions in a reduction sequence move arbitrarily deep.In this paper it is shown that both Convergence and Strong Convergence are modular properties of non-collapsing Infinitary Term Rewriting Systems, provided (for convergence) that the term metrics are granular. This generalises known modularity results beyond metric d∞. Modular properties of Term Rewriting are properties that are preserved under (and reflected by) the disjoint union of signatures. They are of particular interest to reason about rewrite systems by divide and conquer.. Since Toyama showed that confluence is [22] and strong normalisation is not [23] modular, the modularity of many further properties have been investigated. Examples of other modular properties are: uniqueness of normal forms [14] , weak normalisation, simple termination; examples of non-modular properties are: completeness, strong confluence. Modularity proofs for these properties are sometimes complex [22] , sometimes simple: the modularity of weak confluence follows directly from the critical pair lemma [2] . Many modularity results come with syntactic restrictions, e.g. completeness is modular in the absence of non-left-linear rules [21] and strong normalisation in the absence of collapsing rules [16] .. One aspect of Term Rewriting is that it can be viewed as a computational model for Functional Programming. Lazy Functional Programming exhibits a phenomenon that ordinary Term Rewriting does not really capture: reductions that converge to an infinite result in infinitely many steps. Partly for that reason, the concept of Infinitary Term Rewriting was devised in [5] ; technically, it arises from the finite version by equipping the signature with a metric d m and using as its universe of terms Ter m (Σ), the metric completion of the metric space (Ter (Σ ), d m ), where Ter (Σ ) is the set of finite terms.. It is common to only consider the metric d ∞ (which goes back to [1] ) in infinitary term rewriting, ever since its beginning in [5] ; d ∞ sets the distance d ∞ (t, t ′ ) to 2 −k where k is the length of the shortest position at which the two terms t and t ′ have different subterm-roots. This also coincides with a straightforward co-inductive definition of infinitary terms [13] . The metric completion of d ∞ plays a special role as it is a final co-algebra and thus contains all infinite terms.. However, this is not the only metric of interest: for instance, the functional programming language Haskell permits to equip constructors with strictness annotations [8] . These annotations change the universe of terms by ensuring that certain infinite terms are impossible to construct. For example:. data LTree = Null | Bin LTree Int !LTree The exclamation mark before the third argument of the constructor Bin labels that argument as strict. For Haskell, this means that to construct a value for Bin t1 n t2 the expression t2 needs to be evaluated first. As a consequence it is impossible to construct an LTree with an infinite right spine. The same effect can be achieved by modifying the distance function [9] ; in this case, we could define With this, the distance between two finite LTree values would be 1 if their right spines have different lengths. Therefore, all Cauchy-sequences would eventually have right spines of a fixed finite length with the corresponding implication for metric completion. Metrics of this form are exactly the granular metrics mentioned in title and abstract. They distinguish between strict and lazy argument positions, which correspond to multiplicative factors of 1 and 1 2 (respectively) in the components of their associated ultra-metric maps. The associated ultra-metric maps tell us how to compute distances between terms with the same root symbol, and the components split this task into different argument positions. The reasons to treat these metrics separately are (i) the practical interest in them and (ii) that their restricted form is particularly well-behaved.. Infinitary Term Rewriting views transfinite reductions as reduction sequences that are also convergent sequences, w.r.t. its metric. In a reduction sequence each term rewrites to its successor. An infinitary Term Rewriting System (iTRS) is called convergent iff all its reduction sequences converge.. Before [10] there were few positive results about convergence in the literature, instead research has focussed on strong convergence which in addition requires that redex positions in a reduction sequence cannot stay short: if we replace each contracted redex with a random term we would still have a converging sequence.. It is known that strong normalisation is not modular for finitary Term Rewriting Systems. The well-known counter example from [23] has a strongly normalising TRS with rule F (0, 1, x) → F (x, x, x), and another one with rules G(x, y) → x, G(x, y) → y; their combination fails to be strong normalising as the term F (0, 1, G(0, 1)) reduces to itself in three steps. As we shall see, for the discrete metric strong convergence and strong normalisation coincide and therefore modularity of strong convergence can fail for other term metrics than d ∞ . The example also shows that "finite convergence", the convergence of all reduction sequences starting from a finite term, fails to be modular in both weak and strong form [19] .. Simonsen [19] proved that strong convergence is a modular property for left-linear iTRSs (for metric d ∞ ); the left-linearity condition is indeed unnecessary [10] . Proving such results for ordinary convergence is more complicated. Example 1.1. R 1 = {F (x, x, y) → F (x, y, x)} R 2 = {0 → S(0)} Both systems R 1 and R 2 are converging. In the combination we have reductions like F (0, 0, 0) → F (0, 0, 1) → F (0, 1, 0) → F (1, 1, 0) → F (1, 0, 1) → F (1, 1, 1), with 1 = S(0). If we project position 2 (or 3) from this reduction sequence, thus extracting a sequence of elements from the other system then we get a sequence where the values go down as well as up, i.e. on the partial order induced by R 2 : 0, 0, 1, 1, 0, 1, 1, . . . For an overall convergence proof we need to demonstrate why that projected sequence is supposed to be converging, and that is not trivial.. If a reduction sequence in the disjoint union of two iTRSs is converging but not strongly converging then its top layer "remains active" throughout the sequence and in particular it can rearrange its principal subterms, as happened with 0 and 1 in Example 1.1. Informally, principal subterms are those subterms where first a change of signature occurs.. These rearrangements are strongly constrained though by the fact that the iTRS associated with the top layer is itself converging. In particular, it can be shown that the subterms at a principal position in the limit form a so-called focussed sequence; these have peculiar properties: (i) any infinite subsequence of a focussed sequence contains an infinite weak reduction sequence; (ii) any non-converging focussed sequence contains a subsequence which is a non-converging weak reduction sequence. Here, "weak reduction sequences" are reduction sequences with gaps: an element of such a sequence does not necessarily reduce to its neighbour in a single reduction step, this can take any number of steps. For abstract reduction systems over Hausdorff spaces, convergence of all reduction sequences and convergence of all focussed sequences are equivalent properties (Proposition 3.8).. This implies that any reduction sequence in the combined iTRS must converge "layer by layer", and therefore is converging w.r.t. the metric d ∞ . Thus such a sequence can only fail to converge if the infinite term the sequence is converging to (under metric d ∞ ) does not exist in the metric completion derived from the combined system's own metric d m . For example, if one system has the rule F (x) → G(x) and the other a unary function symbol H then the infinite term t = F (H(t)) may exist under the combined metric whilst u = G(H(u)) may not, in which case the combined system would fail to converge. In granular term metrics such rules can be dismissed as they instantly break convergence; in the example, a = F (a) would and b = G(b) would not exist. For the standard metric d ∞ it has been shown that both weak and strong convergence are modular properties of non-collapsing iTRSs. Otherwise, the iTRSs involved can have various usually undesirable properties: non-left-linear rules, infinite left-hand sides, and infinitely many rules are all permitted.. The proof for strong convergence is perfectly straightforward, the proof for weak convergence is not. At the heart of the latter is a consideration that the reductions in the top layer are limited in the way they can rearrange the principal subterms, in particular that the principal subterms in a fixed position must form a focussed sequence. Moreover, abstract reduction systems are converging iff their focussed sequences do, and both observations together give that result.. For other metrics, these arguments still apply to reductions of finite rank (Theorem 9.7), while for reductions of infinite rank there are counterexamples. As a general proof strategy, various convergence arguments have been constructed along the central idea of limiting the pointwise distance between the reduction sequence under investigation and a finite rank approximation of that sequence. These allow to generalise the modularity of convergence and strong convergence of non-collapsing iTRS beyond metric d ∞ , to systems with either granular term metrics or (not elaborated here) arbitrary metrics but depth-preserving rules.

B O B O O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O B O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O B O O O O O O O O O O O O O O O O O O O O O O O O


Fast algorithms for computing the Tripartition-based Distance between Phylogenetic Networks Abstract. Consider two phylogenetic networks N and N ′ of size n. The tripartition-based distance finds the proportion of tripartitions which are not shared by N and N ′ . This distance is proposed by Moret et al (2004) and is a generalization of Robinson-Foulds distance, which is orginally used to compare two phylogenetic trees. This paper gives an O(min{kn log n, n log n + hn})-time algorithm to compute this distance, where h is the number of hybrid nodes in N and N ′ while k is the maximum number of hybrid nodes among all biconnected components in N and N ′ . Note that k << h << n in a phylogenetic network. In addition, we propose algorithms for comparing galled-trees, which are an important, biological meaningful special case of phylogenetic network. We give an O(n)-time algorithm for comparing two galled-trees. We also give an O(n + kh)-time algorithm for comparing a galled-tree with another general network, where h and k are the number of hybrid nodes in the latter network and its biggest biconnected component respectively. Phylogenetic trees are traditionally used to describe evolutionary relationships among a set of objects. However, evolutionary events such as horizontal gene transfer or hybrid speciation (often referred to as recombination events) which suggest convergence between objects cannot be adequately represented in a single tree structure. In the famous Science paper [3] by Doolittle, he also pointed out that the phylogenetic tree is inadequated to represent the "true" evolution history. To solve the shortcoming, phylogenetic networks were introduced. A phylogenetic network is a distinctly leaf-labeled directed acyclic graph where the in-degree and out-degree of all nodes are bounded above by 2. The nodes with in-degree 2 are called hybrid nodes and they are used to model the recombination events. Fig. 1(a) shows an example of a phylogenetic network.. Recently, a lot of works have been proposed to reconstruct phylogenetic networks [1, 4-6, 8-11, 15-17, 19] . To access the topological accuracy of different construction methods, two measurements were proposed for comparing networks. They are Maximum Agreement Phylogenetic Subnetwork (MASN) [2, 13] , and Tripartition-based distance [14] . This paper focuses on the latter measurement.. The tripartition-based distance is a generalization of the Robinson-Foulds measure [18] , which is a well-known method for comparing phylogenetic trees. Given two phylogenetic networks N and N ′ which have the same leaf set, the tripartition-based distance tri(N, N ′ ) computes the proportion of tripartitions (defined in Section 2.3) which are not shared by N and N ′ . The tripartition-based distance is shown to be a distance metric [14] . More importantly, when both N and N ′ are trees, tri(N, N ′ ) equals the Robinson-Foulds distance between them. In this paper, we compute tri(N, N ′ ) in O(min{kn log n, n log n + hn}) time where n = max{V (N ), V (N ′ )}, h is the maximum number of hybrid nodes in N and N ′ , and k is the maximum number of hybrid nodes among all the biconnected components in N and N ′ . As the number of hybrid nodes in a network is relatively rare (recombination events do not happen frequently), k << h << n. Thus, in practice, the running time of our algorithm achieves O(n log n).. We also consider comparing an important, biologically motivated special case of phylogenetic networks, known as galled-trees. A galled-tree [2, 5, 10-13, 16, 19] (also referred to in the literature as a a level-1 network [2, 12] , a gt-network [16] , or a topology with independent recombination events [19] ) is a phylogenetic network in which all cycles in the underlying undirected graph are node-disjoint (see the network N in Fig. 2 for an example). When both N and N ′ are galled-trees, we show that tri(N, N ′ ) can be computed in O(n) time. If only N is known to be a galled-tree, tri(N, N ′ ) can be computed in O(n + kh) where h and k are the number of hybrid nodes in N ′ and in the biggest biconnected component of N ′ respectively. Our improvement is stemmed from a novel labeling technique which labels the nodes of the networks to facilite efficient identification of common tripartitions between two networks.. The rest of the paper is organized as follows. We first present the preliminaries in Section 2. Section 3 details the results for computing tri(N, N ′ ) when at least one of N and N ′ is a galled-tree. Finally, we present the result for computing the tripartition-based distance for two general networks in Section 4.

O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O


Dynamic overlay of scratchpad memory for energy minimization The memory subsystem accounts for a significant portion of the aggregate energy budget of contemporary embedded systems. Moreover, there exists a large potential for optimizing the energy consumption of the memory subsystem. Consequently, novel memories as well as novel algorithms for their efficient utilization are being designed. Scratchpads are known to perform better than caches in terms of power, performance, area and predictability. However, unlike caches they depend upon software allocation techniques for their utilization. In this paper, we present an allocation technique which analyzes the application and inserts instructions to dynamically copy both code segments and variables onto the scratchpad at runtime. We demonstrate that the problem of dynamically overlaying scratchpad is an extension of the Global Register Allocation problem. The overlay problem is solved optimally using ILP formulation techniques. Our approach improves upon the only previously known allocation technique for statically allocating both variables and code segments onto the scratchpad. Experiments report an average reduction of 34% and 18% in the energy consumption and the runtime of the applications, respectively. A minimal increase in code size is also reported.Several researchers [10, 14] have identified the memory subsystem as the energy bottleneck of the entire system. Memory hierarchies are being constructed to reduce the memory subsystem's energy dissipation. Caches and scratchpad memories represent two contrasting memory architectures. Caches improve the performance by exploiting the temporal and spatial locality present in the program. However, for embedded systems, the overheads associated with caches often negate their benefits. Moreover, caches are notorious for their unpredictable behavior [9] .On the other hand, a scratchpad memory (SPM) consists of just a memory array and address decoding circuitry. Due to the absence of the tag memory and the comparators, scratchpad memories require much less energy per access than a cache. In addition, they require less onchip area and allow tighter bounds on WCET prediction of the system. However, they require complex program analysis and explicit support from the compiler.Previous work on SPM usage has mainly focused on static allocation of the SPM. Unlike a cache which adapts its contents according to the program behavior, a statically allocated SPM does not modify its contents during the runtime of the program. The static allocation technique may lead to under utilization of the SPM. Consequently, we present a profile based approach which on the basis of live ranges of both variables and code segments, replenishes the contents of the SPM. The technique identifies the points in the program to insert instructions to copy the contents on and off the SPM. The points are optimally chosen in order to cause the least overhead. The technique also computes addresses within the SPM address range where variables and code segments are to be copied. These addresses are computed such that a large number of variables and code segments share the same SPM space.The rest of the paper is structured as follows: After the presentation of related work, the algorithm for solving the scratchpad overlay problem is described in Section 3. This is followed by a presentation of the experimental workflow. A discussion on the experimental results is presented in Section 5. The paper ends with a conclusion and future work. The design of embedded systems is very much driven by applications. It is expected that future applications will be derived from the multimedia domain and will require significantly more processing power. As a result, powerful processors and large memories will have to be used in future embedded systems. It is evident that these processors and memories are exteremely power-hungry, whereas the electrical energy available in embedded systems (especially in portable systems) is strictly limited. Consequently, a significant research on low power techniques is being performed. In this paper we presented a liveness analysis based technique for dynamic utilization of the scratchpad memory. The problem of dynamically overlaying both data and instructions onto the SPM was shown to be an NP-complete problem. A technique which breaks the scratchpad overlay problem into two smaller problems and obtains an optimal solution for both the problems was presented. The presented technique enables efficient utilization of the SPM and results in reduced energy consumption of the system against a published algorithm. The average reductions in energy consumption and execution time are 34% and 18%, respectively. In the future, we intend to extend the approach to handle inter-procedural overlay of the SPM and would also like to explore the possibility of obtaining near-optimal results using heuristic approaches. The research on scratchpad utilization can be classified into two broad categories viz. static allocation and dynamic allocation techniques. In the former, the SPM is loaded once at the start and its contents remain invariant during the entire execution of the application. On the other hand, dynamic allocation techniques change the SPM contents during the execution to reflect the dynamic behavior of the application.. Most of the research [3, 14] on static allocation of the SPM has focused on assigning data elements. The authors [14] were Figure 1 : Workflow of the Scratchpad Overlay Algorithms the first-ones to demonstrate the effectiveness of the SPMs. They achieved commendable reductions in the runtime of the application by mapping frequently accessed data elements onto the SPM. Authors in [3] proposed an optimal algorithm for assigning data elements onto the SPM. The algorithm is based upon profiling the application and solving a system of binary linear equations. In both the approaches, data elements (e.g. arrays) that are too large to fit in the SPM are kept in the main memory. A recent approach [20] utilizes the SPM to store instructions and models an instruction cache present in the system as a conflict graph. Instructions are mapped to the SPM in order to minimize the conflict cache misses and the energy consumption. Authors in [17, 19] demonstrated the benefits that can be achieved by assigning both instructions and data elements to the SPM.. In contrast, dynamic utilization of the SPM is still a field of ongoing research. Authors of [10] use Presburger formulas to dynamically copy parts of arrays at runtime. Dynamic copying of instructions is proposed by authors in [15] . Certain infrequently executed points (e.g. entry point of a function or a loop) are considered as copy-points. At these copy-points, copying of the following basic blocks is considered in order to minimize the energy consumption.. Global register allocation is one of the most researched and fundamental topics in code optimization and compiler construction [9] . A compiler initially generates code assuming an infinite number of symbolic registers which have to be assigned to the limited number of the processor's real registers. Those symbolic registers which cannot be assigned a real register are kept in the main memory. Spill code is inserted to bring these symbolic registers to and from the main memory. Global register allocation attempts to find an assignment of the symbolic registers to the processor's real registers such that the overhead due to the generated spill code is minimized. The allocation problem was proven to be NP-complete [7] . Consequently, most of the allocators use a graph coloring [5] based heuristic. These allocators perform near-optimal allocation for regular architectures with a large number of processor registers. However, they fail considerably for irregular architectures which invariably come with a smaller number of registers. In the recent past, approaches for optimal register allocation [1, 8, 11] have been proposed. The register allocation problem is formulated as an integer linear program (ILP) and solved using a general purpose ILP solver [6] . Although global register allocation is NP-complete, authors [8] have empirically demonstrated that it takes O(n 3 ) time to compute the optimal register allocation for real-life benchmarks. The runtime was reduced to O(n 1.3 ) when the problem was relaxed to compute near-optimal solutions [1].

B B O B O O B I O O O O O O O O O O B B B O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B B O O O O O B O O O B B O O O O O O B O O O B O O O O O O O O O O O O B O O O O O B O B O B O O O O O O B O O O O O B B I O O B O O O O O B O O O O O O O O O O B O O O O O O O B O O O B O O O O B O O O O O O B I O O B O O B O O B O O B B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O B O B O O O O O O B O O B O O O O O O O O O O O O O B I O O O O B O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O B O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B I O O O O O B O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O O B O O O O O O B O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B O O O O B I O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B B O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B B I O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B B I O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O B O O O O O O B O O O O B B I O O O O O O O O O O O O O O O O O O B I O O O O B O O O O O O O O O O O O O O O O


An architecture for exploiting multi-core processors to parallelize network intrusion prevention It is becoming increasingly difficult to implement effective systems for preventing network attacks, due to the combination of (1) In previous work [13] , we have argued that the performance pressures on implementing effective network security monitoring are growing fiercely in multiple dimensions: (1) attacks continue to improve due to the adversarial nature of network security; (2) the power of simple "signature matching"-looking for specific strings or regular expressions within packets or reassembled byte streams-has drastically dwindled due to the major problems of false positives, polymorphism, and "zero day" attacks; (3) moving beyond signature-matching requires sophisticated analysis of protocols at higher semantic levels, and incorporating context correlated across multiple connections, hosts, sensors, and over time; (4) increasingly, we need to not only analyze traffic but transform it ("normalization" [8] ) to eliminate broad classes of evasion threats, and, even more critically, to realize intrusion prevention systems); and (5) all of this in the presence of ever-increasing traffic volumes and rates.. In addition, in the face of all these performance pressures we have also lost our traditional ace-in-thehole, Moore's Law for uniprocessors. Starting around 2002, the performance scaling curve for single CPUs has slowed precipitously. Over the fifteen prior years, uniprocessor performance increased 50-60% per year. But by 2006, performance was a factor of three slower than had the pre-2002 curve continued.. To perform sophisticated network analysis, it is hugely advantageous if we can draw upon the flexibility and inexpensive system costs of using general-purpose CPUs rather than custom hardware such as FPGAs and ASICs. Recently, hardware vendors have begun delivering commodity CPUs that again reflect Moore's Law-style scaling-but with the parallelization gains coming from multi-core/multi-thread architectures. However, while the aggregated throughput of such processors does in fact still follow Moore's law, to exploit the full power of these architecture we must explicitly structure our applications in a highly parallel fashion, dividing the processing into concurrent tasks while minimizing inter-task communication.. Taking advantage of the full power of multi-core processors requires an in-depth approach in order to realize speedups for sophisticated analyses that require finegrained coordination between concurrent threads. First, to provide intrusion prevention functionality (i.e., active blocking of malicious traffic), we must ensure that packets are only forwarded if all relevant processing gives approval. Second, to perform global analysis (e.g., worm contact graphs [6] or content sifting [18] ) we must support exchange of state across threads, but we must minimize such inter-thread communication to maximize performance. Similarly, we must understand how the memory locality of different forms of analysis interacts with the ways in which caches are shared across threads within a CPU core and across cores. We need to be able to express the analysis in a form that is independent to the memory and threading parameters of a given CPU, so we can automatically retarget the implementations of analysis algorithms to different configurations. Finally, we must ensure that our approach is amenable to analysis by performance debugging tools that can illuminate the presence of execution bottlenecks such as those due to memory or messaging patterns.. In this work we frame an architecture customized for parallel execution of network attack analysis. The goal is to support the construction of highly parallel, inline network intrusion prevention systems that can fully exploit the power of modern and future commodity hardware. We first discuss related work, including the large potential of parallel processing for network security analysis ( § 2). We then sketch a high-level overview of our architecture ( § 3) , and explore what pursuing it would mean in more concrete terms ( § 4), before briefly summarizing ( § 5). The goal of our effort is to develop a framework to support the construction of highly parallel, inline network intrusion prevention systems that can fully exploit the power of modern and future commodity hardware. Ultimately, we aim to enable network intrusion prevention to reap both the benefits of executing on general-purpose CPUs, and the exponential scaling that Moore's Law for aggregate parallel processing continues to promise.. The key elements of achieving this vision are (i) identifying the optimal thread granularity for a given hardware architecture so we can structure the data-flow accordingly, (ii) devising scalable inter-thread communication schemes, (iii) resolving intrusion-prevention go/no-go decisions in a timely and reliable fashion, and (iv) supporting effective evaluation, profiling and debugging of such systems. We have sketched how we believe we can achieve all of these by employing a custom frontend component (the Active Network Interface) and structuring our network analysis in an event-oriented fashion.. If successful, our approach will offer a design point in terms of price and ease of scalability that significantly differs from that offered by current approaches based on FPGA, ASIC, or network processor hardware. Given how parallel hardware appears likely to evolve in the future, this paradigm could ultimately prove highly influential to industry. Parallelizing analysis. To date, efforts on exploiting parallelism for network security monitoring have focused heavily on signature scanning, i.e., detecting whether a packet (or sometimes a reassembled byte stream) contains a string of interest or matches a regular expression, and executing an action (such as drop or alert) associated with the signature. Much of this work has drawn inspiration from the popularity of "Snort" [16] and its large set of byte-level signatures. This work includes use of nondeterministic finite automata to match regular expressions [17] , compiling regular expressions into deterministic finite automata [12] , building optimized Aho-Corasick trees for sets of strings [21] , and specialized architectures based on collections of highly optimized tiny state-machines [20] .. A vital point regarding much of the previous parallel hardware design research is that it presumes a nearly stateless approach to attack detection. The systems either operate on single packets or assume that a separate process reassembles the TCP byte stream. Parallelizing richer, stateful hardware elements, such as TCP stream reassembly, have not been explored in as much depth; see our previous work in [4] and the discussion of prior efforts therein, including the vulnerability to such TCP processors to evasion attacks [15, 8] .. In terms of parallelizing higher-level network security analysis, we take our main inspiration from the work of Kruegel et al, who explored the design of front-end NIDS load balancers [10] . They introduced the notion of slicing: splitting up traffic not simply at a per-connection granularity, but in a NIDS-analysis-aware fashion to ensure that packets germane to possible attack scenarios are all available to the processing element that assesses their associated scenarios.. However, the issue of such front-end dispatch becomes subtle because of the many forms of global analysis. For example, content sifting [18] requires looking at a large pool of potentially suspicious strings that may be taken from any connection, and contact graph analysis [6] can efficiently detect new worms, but requires a global connection history within a time window. Many attacks seen today involve complex application-level sessions that span multiple connections and sometimes multiple hosts.. Thus, for in-line intrusion prevention operation, we need to go significantly beyond Kruegel's slicing approach to also incorporate (i) ways of structuring the analysis itself such that it is amenable to multi-core parallelization, and (ii) support for prevention (blocking) functionality.. Modern general-purpose CPUs. By relying on customized hardware rather than general-purpose CPUs, commercial systems have difficulty in tracking Moore's Law-style scaling, due to the low level at which parallelism must be expressed (FPGA and ASIC designs) or weak memory caching semantics (network processors).. Modern CPU designs include symmetric multithreaded CPU cores [1, 11] , which allow a single CPU to switch between multiple independent threads of execution, and multi-core systems, where a single die holds multiple CPUs [2, 11] . Recent systems support both: multiple CPUs each executing multiple threads.. It is critical to recognize that to exploit the power of such processors, programs must be specifically designed to have a parallelizable structure. However, when developing software for these systems, not only is it crucial to parallelize the program's execution structure, but also its memory access patterns. Although multi-thread/multicore CPUs preserve the semantics of shared memory with cache-coherence, memory locality and behavior can completely dominate a program's ultimate performance.. In a multi-threaded core, the threads must share a common working set, lest thrashing significantly degrade performance [9] . In contrast, on a multi-core system having disjoint working sets on different cores can be a benefit, as the L1 and often also the L2 caches are independent. When coupled with independent memory controllers [3] , it becomes vital to create and feed the threads in a memory-aware manner.. 3 Overview of the Architecture Figure 1 illustrates the overall structure of our architecture. At the bottom of the diagram is the "Active Network Interface" (ANI). This component provides the inline interface to the network, reading in packets and later (after they have been approved) forwarding them. It also serves as the front-end for dispatching copies of the packets to the analysis components executing on different cores/threads.. The ANI drives its dispatch decisions based on a large connection table indexed by packet header five-tuple. The table yields a routing decision for each packet: either (i) which thread will analyze the packet, (ii) that the ANI should drop the packet directly without further processing, or (iii) that the ANI should forward the packet directly (to enable some forms of off-loading, as discussed below). There is an analogous table indexed by IP addresses to provide per-host blocking, and also default routing for packets not found in either table.. The analysis components populate the ANI's table entries to control its dispatch procedure. For example, a component can install a drop action to cut off a misbehaving connection, or alter the thread associated with a connection for purposes to improve locality of reference.. The ANI dispatches packets for analysis by writing them into queues in memory associated with the thread (and core) assigned to analyze the corresponding flow. It also sends a corresponding descriptor used to subsequently refer to the packets. The ANI holds copies of the packets locally pending approval to forward them, which an analysis component can signal by sending a control message that includes the descriptor back to the ANI. 1 As shown by the solid line from CPU Core 1 to the ANI in the figure, the analysis components can also rewrite pending packets. This functionality is necessary to support normalization, which may require altering the contents of packets [8] .. Conceptually, the packet queues reside in the processor's shared memory. In general, these writes will directly target the processor's shared L2 cache. On modern multi-core systems, such a write will invalidate the L1 cache entries local to the individual cores, enabling the threads executing in that core to detect that they have a new packet waiting for them and load it from L2 cache to L1 cache.. An important point is that unlike for the rest of the architecture, we make the presumption that the ANI is custom hardware, specialized for the task. Our recent work has shown that we can construct such hardware efficiently and affordably using a simple FPGA design [24] . There are already at least two suitable Gbps Ethernet FPGA platforms available, the four-port GigE copper NetFPGA [23] and the 6 SPF (fiber) port HyperTransport-based HTX board [7] .. We structure the analysis components as an eventbased system, which we have developed in previous work as offering great power for network security analysis [14] . Doing so allows us to find many opportunities for concurrent execution, since events introduce a natural, decoupled asynchrony into the flow of analysis. By associating events with the packets that ultimately stimulated them, we can determine when all analysis for a given packet has completed, and thus whether it is safe to forward the pending packet.. Parallelizing event execution requires care, however. First, temporal relationships exist between events, which means that their their subsequent handlers cannot execute in arbitrary order. Second, event handlers tend to share a large amount of state, and thus need to access the same memory, potentially blocking execution of other threads. Our architecture envisions addressing these issues by introducing multiple event queues which collect together semantically related events for FIFO execution. Because the events are related, keeping them within a single queue localizes memory access to shared state. This in turn allows for efficient threaded execution of events since the threads can efficiently communicate (and lock data structures, when necessary) by exploiting the per-core memory caches.. The analysis proceeds in stages. The initial stages concern low-level tasks such as TCP stream reassembly and normalization, suitable to a single thread of execution. This stage requires very little inter-thread communication. It outputs events parameterized with parsed packet headers and payload byte streams. The next stage performs application-layer protocol parsing. The outputs from this stage are events reflecting application-level control information (requests and responses) with associated ADUs. Finally, these events are consumed by multiple high-level analyzers that detect attacks both within Table Host Table Core 2 application dialogs and across multiple connections and hosts. In the figure, each core has two queues associated with it, one for receiving packets from the ANI and one for managing the events that its analysis generates and consumes. (Sharing queues across all of a core's threads minimizes potential thrashing of the limited L1 cache.) Communication between threads occurs either via the shared memory or by passing events. Events exchanged between threads executing in the same core generally use the core's event queue, while communication across cores can use separate per-core queues (e.g., "Core 1 MSG-Event-Q" in the figure) . The figure shows Core 1 inserting elements into the queue for Core 2, and reading from its own MSG-Event-Q. Similarly, the system can receive externally generated events (e.g., from a host-based IDS) and send events to external agents (e.g., a global management console such as HP OpenView) via "External MSG-Event-Q".

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


A framework for designing benchmarks of investigating digital forensics tools for mobile devices With the continued growth of the mobile device market, the possibility of their use in criminal activity will only continue to increase. While the mobile device market provides a great variety of manufacturers and models causing a strong diversity, it becomes difficult for a professional investigator to choose the proper forensics tools for seizing internal data from mobile devices. Through this paper, we will give a comprehensive perspective of each popular digital forensic tool and initiate discussion towards benchmarking key features and considerations of forensic analysis techniques. There are only few benchmarks or standards set for digital investigations involving traditional computers and general guidelines and accepted best practices for network forensics. However, there aren't any for mobile forensics because they are still relatively new and still considered emerging fields in research and industry. Computer forensics precedents are just now being converted to standards to try and accommodate criminal investigations. Traditional computer forensics can be traced back as far as 1993 when the US Supreme Court Case, Daubert v. Merrill Dow Pharmaceuticals, set a precedent regarding the admissibility of digital forensic evidence. Courts were given discretion whether to accept scientific evidence based on the following criteria [2] :. The theory or technique utilized must have been tested and that test must be replicable. The theory or technique must have been subject to peer review and publication. The error rate associated with the technique must be known. The theory or technique must enjoy general acceptance within the scientific community Earlier investigations where digital evidence was involved were categorized with traditional forensic science were governed by the Frye principles, where evidence gathered using "generally accepted practices" were admissible in court. As recently as 2000, both Frye and Daubert principles were revised, updated, and released as the Federal Rules of Evidence (FRE). [2] Although similar in nature and concept to its desktop counterpart, mobile devices operate drastically different. Even though there are only five mobile operating systems that dominate the mobile device market, there are a wide variety of device manufacturers creating many more variants. The variants make it improbable to create a common, reproducible set of data to test for benchmarking and standardizing digital forensic techniques. [4] In this paper, we attempt to address those issues with designed benchmarks for specific operating systems.. There are a few ways that can be discussed in creating benchmarks for mobile forensics, for example by forensic tool or by operating system and in some cases device model. This case study will use forensic tools and mobile device operating systems as benchmarks categories. The forensic tools will encompass benchmarks for mobile device type, mobile device model, and time it takes to perform an acquisition while the mobile device operating systems category will consists of benchmarks for device models and network carrier. While we have established the benchmark categories of forensic tools and operating systems, the benchmarks themselves will be created according to the data types received from the forensic acquisitions during experimentation.. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. 49th ACM Southeast Conference, March 24-26, 2011, Kennesaw, GA, USA. Copyright 2011 ACM 978-1-4503-0686-7/11/03…$10.00. This paper is organized as follows: section 2 will discuss the forensic analysis of mobile devices, while section 3 will discuss tools available for forensics of mobile devices. Section 4 will discuss related work; section 5 will discuss proposed work and this case study will be carried out, followed by conclusion in section 6. Law enforcement agencies want to process all evidence as quickly and efficiently as possible, while also getting all important information for their investigation. Through this case study, we hope to start the process of creating some benchmarks for mobile forensics by modifying those already in place for computer & network forensics and adapting them to the mobile environment that we live in.. We would make the set of benchmarks available for any researcher who wants to compare the new tools with other tools for different operating systems. In the future we hope to improve on existing benchmarks and continuously retrieve various feedbacks to make benchmarks more effective and easy to use. A literature review found little to no papers, journal articles, or conference proceedings that focused on forensic analysis of more than two operating systems or forensic tools. Some standards and benchmarks available to traditional computer forensics were briefly mentioned in the introduction. In an evaluation of CFTL [9] , the introduction of general benchmarks for timestamp-based digital forensics was discussed. NIST recently published another study, in October 2009, which revised standards for cell phone forensics using device model, SIM, and removable memory as benchmarks.

O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O


On the quest for optimal rule learning heuristics The primary goal of the research reported in this paper is to identify what criteria are responsible for the good performance of a heuristic rule evaluation function in a greedy top-down covering algorithm. We first argue that search heuristics for inductive rule learning algorithms typically trade off consistency and coverage, and we investigate this trade-off by determining optimal parameter settings for five different parametrized heuristics. In order to avoid biasing our study by known functional families, we also investigate the potential of using metalearning for obtaining alternative rule learning heuristics. The key results of this experimental study are not only practical default values for commonly used heuristics and a broad comparative evaluation of known and novel rule learning heuristics, but we also gain theoretical insights into factors that are responsible for a good performance. For example, we observe that consistency should be weighted more heavily than coverage, presumably because a lack of coverage can later be corrected by learning additional rules. The long-term goal of our research is to understand the properties of rule learning heuristics, that will allow them to perform well in a wide variety of datasets. Although different classification rule learning algorithms use different heuristics, there has not been much work on trying to characterize their behavior. Notable exceptions include Lavrač et al. (1999) , which proposed weighted relative accuracy as a novel heuristic, and Fürnkranz and Flach (2005) , in which a wide variety of rule evaluation metrics were analyzed and compared by visualizing their behavior in ROC space. There are also some works on comparing properties of association rule evaluation measures (e.g., Tan et al. 2002) but these have different requirements than classification rules (e.g., completeness is not an issue there).. In this paper, we will try to approach this problem empirically. We will first empirically compare and analyze a number of known rule learning heuristics. Rule learning heuristics, in one way or another, trade off consistency and coverage. On the one hand, rules should be as consistent as possible by only covering a small percentage of negative examples. On the other hand, rules with a high coverage tend to be more reliable, even though they might be less precise on the training examples than alternative rules with lower coverage. An increase in coverage of a rule typically goes hand-in-hand with a decrease in consistency, and vice versa. In fact, the conventional top-down hill-climbing search for single rules follows exactly this principle: starting with the empty rule, conditions are greedily added, thereby decreasing coverage but increasing consistency.. In this work, we will show that five well-known rule evaluation metrics (a cost tradeoff, a relative cost trade-off, the m-estimate, the F -measure, and the Klösgen measures) provide parameters that allow to control this trade-off. In an extensive experimental studyto our knowledge the largest empirical comparison of rule learning heuristics to datewe aimed at determining optimal values for each of their respective parameters. We will compare these settings to standard heuristics and show that the new settings outperform the fixed consistency/coverage trade-offs that are commonly used as rule learning heuristics. By testing the performance of the optimized heuristics on an additional selection of datasets not used for optimization, we will ensure that this performance gain is not due to overfitting the training datasets.. However, optimizing parameters constrains the candidate heuristics to known functional shapes. Consequently, we will then try to leave these constraints behind and try to discover entirely new heuristics. The key idea is to meta-learn such a heuristic from experience, without a bias towards existing measures. Consequently, we created a large meta dataset (containing information from which we assume that the "true" performance of a rule can be learned) and use various regression methods to learn to predict this performance. On this dataset, we learn an evaluation function and use it as a search heuristic inside our implementation of a simple rule learner. We report on the results of our experiments with various options for generating the meta datasets, with different feature sets and different metalearning algorithms. In particular, we try to evaluate the importance of rule length as an additional feature and consider a delayed-reward scenario where the learner tries to predict the performance of the completed rule from its incomplete current state in the search space.. The paper is organized as follows: we start with a brief recapitulation of separate-andconquer learning and describe our simple ruler learner, which is used for generating the meta data and for evaluating the learned heuristics (Sect. 2). Section 3 then provides a survey of the heuristics that are experimentally compared in this paper. In this section, we also briefly recapitulate the use of coverage space isometrics for visualizing the preference structure of rule learning heuristics. After a brief description of the experimental setup that will be used throughout the paper (Sect. 4), the main part of the paper describes our experimental work in optimizing known heuristics (Sect. 5) and metalearning new heuristics (Sect. 6). We put the results in perspective by discussing which problems need to be addressed via rule learning heuristics, and which of them are addressed in this work (Sect. 7). The paper is wrapped up with a brief discussion of related work (Sect. 8) and a summary of the most important conclusions drawn from this study (Sect. 9) .. Parts of this paper have previously appeared in Janssen and Fürnkranz (2008) and Janssen and Fürnkranz (2007) . It is also interesting to compare the results of the absolute and relative cost measures: although, as we have stated above, the two are equivalent in the sense that for each individual dataset, one can be transformed into each other by picking an appropriate cost factor, the relative cost measure has a clearly better peak performance exceeding 85%. Thus, it seems to be quite important to incorporate the class distribution P /(P + N) into the evaluation metric. This is also confirmed by the results of the m-estimate and the Klösgen measures. The experimental study reported in this paper has provided several important insights into the behavior of greedy inductive rule learning algorithms.. First, we think that this has been the most exhaustive experimental comparison of different rule learning heuristics to date. We tested five parameter-free heuristics, five parametrized heuristics with a large number of parameterizations, and several different metalearning scenarios. The results confirm several previously known findings (e.g., precision and Laplace overfit, whereas accuracy and weighted relative accuracy over-generalize), but also yielded new insights into their comparative performance. In particular, we have determined suitable default values for commonly used parametrized evaluation metrics such as the m-estimate. This is of considerable practical importance, as we showed that these new values outperformed conventional search heuristics and performed comparably to the Ripper rule learning algorithm. On the other hand, however, we have also seen some indication that these values capture aspects of our algorithm that may not fully transfer to other rule learning algorithms.. Second, our results also let us draw important conclusions about what factors influence a good performance of a rule learning heuristic. For example, we found that heuristics which take the a priori class distribution into account (e.g., by evaluate relative coverage instead of absolute coverage) will in general outperform heuristics that ignore the class distribution (e.g., the F -measure which trades off recall and precision). This is also confirmed by the high weight that this parameter receives in our meta-learned heuristics. Gain heuristics, which take this one step further by considering the distribution of the predecessor rule as a prior class distribution for each individual addition to a rule, may be even more preferable, but this still needs to be investigated more thoroughly.. We also found that for a good overall performance, it is necessary to prefer consistency over coverage, i.e., to weight the false positive rate more heavily than the true positive rate. We can most clearly observe this bias towards minimizing the false positive rate in the optimal parameter value for the relative cost metric, but it can also be observed in other wellperforming heuristics whose isometrics have a very steep slope in the important regions. In the experiments with metalearning and in the good performance of the correlation heuristic we can also observe that heuristics perform better if they increase the emphasis on this aspect for rules with high consistency.. This result may also be interpreted as evidence that a good heuristic has to adapt to the characteristics of the algorithm in which it is used. In our case, this bias towards consistency seems to be a desirable property for a heuristic that is used in a covering algorithm, where incompleteness (not covering all positive examples) is less severe than inconsistency (covering some negative examples), because incompleteness can be corrected by subsequent rules, whereas inconsistency cannot (at least not in a concept learning scenario). This dependency on the dynamics of the algorithm is also confirmed by one of the results of the metalearning study, in which we observed that training on the test-set performance of the candidate rule is somewhat less efficient than training on the performance of its best refinement. Finally, the results of the transfer of the heuristics to other rule learning implementations also seem to confirm that they capture individual characteristics of the algorithm.. However, our results also have their limitations. For example, we have only evaluated the overall performance over a wide variety of datasets. Obviously, we can expect a better performance if the parameter values are tuned to each individual dataset. We think that the good performance of Ripper is due to the flexibility of post-pruning, which allows to adjust the level of generality of a rule to the characteristic of a particular dataset. We have deliberately ignored the possibility of pruning for this set of experiments, because our goal was to gain a principal understanding of what constitutes a good rule evaluation metric for separate-and-conquer learning. We are currently investigating the interplay of pruning and learning in more detail. The crucial step for the performance of a greedy covering algorithm is the choice of the rule evaluation heuristic. In this paper, we have tried to empirically determine a good trade-off between consistency and completeness, which are the most important criteria for evaluating the quality of a rule.. However, it is important to keep in mind that a good rule must optimize or trade off various criteria simultaneously. Among them are:. Consistency: How many negative examples are covered?. In concept learning or decision list learning, if a rule covers negative examples, these misclassifications cannot be corrected with subsequent rules. When sets of unordered rules are learned for multi-class problems, such corrections are possible, but obviously one should nevertheless try to find pure rules.. Completeness: How many positive examples are covered? Even though subsequent rules may cover examples that the current rule leaves uncovered, rule with higher coverage are typically preferred because they bring the learner closer to the goal of covering all positive examples.. Gain: How good is the rule in comparison to other rules (e.g., default rule, predecessor rules)? A rule with high consistency may be bad if its predecessor had a higher consistency and vice versa. Thus, many heuristics, such as weighted relative accuracy, or information gain relate the quality of a rule to other rules.. Utility: How useful will the rule be in the context of the other rules in the theory? A rule with high consistency and completeness may nevertheless be a bad addition to the current theory if it does not explain any new examples.. Bias: How will the quality estimate change on new examples? It is well-known that estimates obtained on the training data will be optimistically biased. A good heuristic has to address this problem so that the algorithm will not overfit the training examples.. Potential: How close is the current rule to a good rule? An incomplete rule, i.e., a rule that is encountered during the search for a good rule, should not be evaluated by its ability to discriminate between positive and negative examples, but by its potential to be refined into such a rule.. Simplicity: How complex or comprehensible is the rule? In addition to its predictive quality, rules are often also assessed by their comprehensibility, because this is one of the key factors for preferring rule learning algorithms over competing inductive classification algorithms. As comprehensibility is difficult to measure, it is often equated with simplicity or rule length.. Our experiments addressed many of this issues directly (consistency, completeness, gain, bias). Some of these criteria are also addressed, at least to some extent, algorithmically. Utility, for example, is addressed in part by the covering loop which removes examples that have been covered by previous rules. Thus, the context of past rules is taken into account. However, it is harder to take the context of rules that will be subsequently learned into account. This is particularly the case because most rule learning heuristics only focus on the examples covered by the rule, and not on the examples that are not covered by a rule. This is contrary to decision tree learning heuristics, which consider all possible outcomes of a condition simultaneously. 5 The Part (Frank and Witten 1998 ) algorithm may be viewed as an attempt to use the best of both worlds. Similarly, Ripper's global optimization phase, where rules in a final theory are tentatively re-learned in the context of all previous and all subsequent rules, may be viewed as an attempt to address this issue.. It is also important to realize that these criteria are not independent. For example, comprehensibility and simplicity are correlated with completeness: simple rules tend to be more general and to cover more examples. Thus, a bias for completeness will automatically correlate with a bias for shorter rules. Similarly, the idea of the Laplace-correction as introduced in CN2 (Clark and Boswell 1991) was to on the one hand correct a too strong bias for consistency over completeness (by, e.g., penalizing pure rules that cover only single examples), and on the other hand also to try to provide more accurate probability estimates.. Nevertheless, it is not clear whether all these points can or should be addressed simultaneously with a single rule learning heuristic. Answering this question is beyond the scope of this paper (in fact, we do believe that these problems should be separated and addressed individually). However, it is the case that common rule learning algorithms essentially assume that all these objectives can be captured into a single heuristic function (only overfitting is frequently addressed by using a separate criterion). Thus, it is a valid and interesting question how good a greedy rule learner can get under this assumption. The work reported in this paper may be viewed as a contribution to answering this question. It is, in our opinion, the necessary first step to a systematic investigation of heuristic rule learning. While there are several empirical comparisons of splitting heuristics for decision tree induction (Mingers 1989; Buntine and Niblett 1992) , there are, somewhat surprisingly, relatively few works that empirically compare different rule learning heuristics. For example, Lavrač et al. (1992a Lavrač et al. ( , 1992b compare several heuristics for inductive logic programming. Most works only perform a fairly limited comparison, which typically introduces a new heuristic and compares it to the heuristic used in an existing system. A typical example for work in this area is (Todorovski et al. 2000) , where the performance of weighted relative accuracy was compared to the performance of CN2's Laplace-heuristic. To our knowledge, our work reported in this paper is the most exhaustive empirical work in this respect.. On the other hand, considerable progress has been made in the principal understanding of rule learning heuristics. As discussed in Sect. 3.1, Fürnkranz and Flach (2005) have introduced coverage space isometrics as a means for visualizing rule evaluation metrics. Using this tool, they have derived several interesting results, such as that the m-estimate effectively trades off precision and weighted relative accuracy. While their paper contributed to a better understanding of rule learning heuristics, the authors concluded that, in general, rule learning heuristics are not yet well understood.. There has also been significant progress on analyzing rule evaluation metrics that are commonly used in descriptive induction tasks such as association rule discovery or subgroup discovery. Most notably, Tan et al. (2002) have surveyed 21 rule learning heuristics and compared them according to a set of desirable properties. In general, they conclude that the choice of the right interestingness measure is application-dependent, but they also identify situations in which many measures are highly correlated with each other. Bayardo Jr. and Agrawal (1999) analyze several heuristics in support and confidence space, and show that the optimal rules according to many criteria lie on the so-called support/confidence border, the set of rules that have maximum or minimum confidence for a given support level. Recently, Wu et al. (2007) showed that a group of so-called null-invariant measures (measures that are not influenced by the number of records that do not match the pattern) can be generalized into a single parametrized heuristic. We plan to analyze this parametrized heuristic with the apparatus that we have used for our results in Sect. 5.. Naturally, there are some similarities between heuristics used for descriptive and for predictive tasks. For example, Lavrač et al. (1999) derived weighted relative accuracy in an attempt to unify these two realms, or Fürnkranz and Flach (2004) analyzed filtering and stopping heuristics and showed that Foil's information gain search and MDL-based pruning has a quite similar effect as support and confidence thresholds that are commonly used in association rule discovery. Nevertheless, it is important to note that good heuristics for descriptive induction are not necessarily well-suited for predictive induction (weighted relative accuracy is a good example). The key difference is that in the latter case one typically needs to learn an entire rule set, where lack of coverage in individual rules can be corrected by the entire ensemble of rules. Inconsistencies, on the other hand, cannot be corrected by the induction of additional rules (at least not in the case of concept learning). In this light, the result of this paper, that good heuristics for predictive induction will favor consistency over coverage, appears to be reasonable.. Our results may also be viewed in the context of trying to correct overly optimistic training error estimates (resubstitution estimates). In particular, in some of our experiments, we try to directly predict the out-of-sample precision of a rule. This problem has been studied theoretically by Scheffer (2005) and Mozina et al. (2006) . In other works, it has been addressed empirically. For example Vapnik et al. (1994) have used empirical data to measure the VC-Dimension of learning machines. also creates meta data in a quite similar way, and tries to fit various functions to the data. But the focus there is the analysis of the obtained predictions for out-of-sample precision, which is not the key issue in our experiments. The primary goal of our experimental work is to determine search heuristics that are optimal in the sense that they will result in the best overall performance on a wide variety of datasets.. Thus, we have to keep several things in mind. First, our results should be valid for a wide variety of datasets with different characteristics. Second, we have to be careful not to overfit the selected datasets. Finally, we have to select ways for assessing the performance of a heuristic. In this section, we will describe our choices for addressing these concerns.

O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I I O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O B O O O O O O B O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O