How crystals that sense and respond to their environments could evolve Abstract An enduring mystery in biology is how a physical entity simple enough to have arisen spontaneously could have evolved into the complex life seen on Earth today. CairnsSmith has proposed that life might have originated in clays which stored genomes consisting of an arrangement of crystal monomers that was replicated during growth. While a clay genome is simple enough to have conceivably arisen spontaneously, it is not obvious how it might have produced more complex forms as a result of evolution. Here, we examine this possibility in the tile assembly model, a generalized model of crystal growth that has been used to study the self-assembly of DNA tiles. We describe hypothetical crystals for which evolution of complex crystal sequences is driven by the scarceness of resources required for growth. We show how, under certain circumstances, crystal growth that performs computation can predict which resources are abundant. In such cases, crystals executing programs that make these predictions most accurately will grow fastest. Since crystals can perform universal computation, the complexity of computation that can be used to optimize growth is unbounded. To the extent that lessons derived from the tile assembly model might be applicable to mineral crystals, our results suggest that resource scarcity could conceivably have provided the evolutionary pressures necessary to produce complex clay genomes that sense and respond to changes in their environment. Developments in DNA computing have shown that computation can be embedded in crystal growth processes (Winfree 1996; Rothemund et al. 2004) , with potential applications to combinatorial search problems (Lagoudakis and LaBean 2000; Mao et al. 2000) and to fabrication tasks in nanotechnology (Cook et al. 2004; Barish et al. 2005) . But does crystal computation have any relevance to what we observe in nature? We speculate here about a possible connection to the origin of life.. The background for this argument is a hypothesis, proposed and developed by Graham Cairns-Smith (Cairns-Smith 1966) , that the first primitive ''organisms'' were clay crystals. In this theory, information (the first ''genes'') consisted of patterns stored as variations in crystal structure that could be propagated during crystal growth. For example, in some layered silicate clays, there are two distinct layer types that appear in a cross-section as a sequence that could be considered the crystal's genotype. Replication would occur by periodic physically-induced fragmentation of crystals into smaller pieces containing the same genotype, leading to exponential increase in the number of organisms. Cairns-Smith considered several types of selective pressures that could have been present and would have resulted in favoring the growth of crystals with non-trivial genotypes. For example, the structure of a layer sequence could result in different crystal morphologies and different susceptibilities to fragmentation. He further envisioned the clays interacting with organic molecules, somehow leading to novel clay-produced organic molecules and eventually to the genetic takeover of organic life forms (Cairns-Smith 1982) .. One of the strengths of Cairns-Smith's hypothesis is that it is easy to see how Darwinian evolution could have gotten started by geological processes. However, while clays are known to interact with organic molecules (Pitsch et al. 1995; Hanczyc et al. 2003) , it is hard to know whether these interactions could have provided evolutionary pressure toward increasing complexity. It is therefore interesting to ask whether there are other possible mechanisms that could have stimulated the original evolution of complex sequence information. In fact, it is hard to envision how anything so simple that it could have arisen spontaneously could have evolved into the remarkable complexity of form and function found in modern biological organisms. The capacity of evolutionary systems to create increasingly complex forms with increasingly adapted function-so-called open-ended evolution-remains poorly understood. To our knowledge, there are no examples in artificial life (Bedau et al. 2000) , in vitro chemical evolution (Wright and Joyce 1997; Joyce 2004) , or in vivo directed evolution (Yokobayashi et al. 2002) that have convincingly demonstrated open-ended evolution.. The conceptual and physical simplicity of crystal evolution makes it a promising place to examine these issues. In this paper, we consider whether there are properties of crystal growth that can lead to open-ended evolution. Examining the capacity for interesting evolutionary landscapes requires distinguishing between crystal genotype and phenotypewhat is the genetically-determined function performed by crystals that gives rise to a selective advantage? In this paper, we investigate the ability of crystals to respond to selection pressures using computations that occur during growth.. The notion that a crystal can perform a computation is based on the observation that the tiling problem (the question of whether a set of geometric shapes can tile the plane) is undecidable: any problem solvable by a Turing machine can be expressed as a question of whether a particular set of shapes can tile the plane (Wang 1962) . This observation led to a constructive method of computing by arranging tiles into a lattice (Winfree 1996) . These tiles can be viewed as analogues for crystal monomers; attachment at a specific site in a growing crystal is determined by how well a tile's shape fits in the growth site. The binding of a particular tile at a particular site can be viewed as a computational or information transfer step.. It is reasonable to assume that crystals grow in environments where their monomers are present in solution at different (possibly time-varying) concentrations. In such an environment, which crystal patterns grow the fastest? Are there environments in which crystals must perform computations in order to grow quickly? We show that crystals can sense the environment and respond by making use of the most abundant tiles. We call this feature a ''crystal metabolism'' because the computation they perform controls the extraction of resources from the environment. In particular, we are interested in whether there are environments that lead to the evolution of increasingly complex crystal metabolisms.. We examine these issues in principle within a previously described tile assembly model (Winfree 1998) , which is a generalized crystal growth model that has been used to study algorithmic self-assembly of synthetic DNA tiles (Rothemund and Winfree 2000; Rothemund et al. 2004; Barish et al. 2005) . It has been previously argued (Schulman and Winfree 2005 ) that DNA tile crystals have the capacity for Darwinian evolution of the sort imagined by Cairns-Smith. Insights derived from studying this model should be applicable to crystals composed of a variety of other materials, such as proteins (Fygenson et al. 1995; Collins et al. 2004) , RNA (Chworos et al. 2004) , or even macroscopic tiles (Bowden et al. 1997; Rothemund 2000) or clay minerals (Cairns-Smith and Hartman 1986) .. The tile assembly model describes crystal monomers as square or rectangular tiles with each unit edge labeled to indicate how it fits with other monomers. Crystal growth proceeds by accretion, with single tiles being added to the crystal at sites where they make a sufficient number of contacts. In this paper we consider a version of this model in which (a) a tile may be added to a site if labels on at least two edges match those presented by the crystal at that site, and (b) monomer tiles arrive at potential binding sites with a frequency proportional to their concentration in solution. Occasional violations of rule (a) are referred to as ''mutations''. A system consists of a set of tiles, along with the concentrations of those tiles in solution. Because of the particular choice of matching rules, each tile set implicitly determines what arrangements of tiles can grow as crystals. If certain arrangements grow faster than others under given conditions, then we consider these arrangements more fit. Later we will discuss how growth rates relate to the rate of exponential reproduction.. To discuss evolution, it is helpful to consider three aspects of an evolutionary process. First, a self-replicating entity carries information which directs behavior. The space of achievable behaviors is therefore inherently dependent on the material from which they are constructed. Second, there must be an environment in which certain behaviors have selective advantage; this provides the stimulus for evolution to discover complex solutions. Third, for evolution to proceed quickly there must be a route via mutations in which ever more complex behavior is achieved by a series of incremental steps. In crystal evolution, the first aspect (potential for complexity) derives from a choice of a particular tile set; this determines the behavior implicit in the crystal growth. The second aspect (stimulus for complexity) derives from the conditions in which the crystals are grown, e.g., tile concentrations and temperature, etc. The third aspect (the route to complexity) is difficult to predict. We will not address it here; for our purposes, it is enough to know that the fittest crystals could arise through (possibly extremely unlikely) mutations or spontaneous generation. However, understanding the route to complexity is an important goal for future studies of crystal evolution.. In addition to the above, open-ended evolution seems to require that the selective pressure provided by the environment is distinct from the potential for evolution, in the sense that different environments must lead to distinct functional solutions. For example, in modern biology, the universal potential of biochemistry-the ability of DNA to code for proteins and other macromolecules that create seemingly arbitrarily sophisticated and complex machines-makes it possible for living organisms to adapt to an incredible variety of environmental niches, opening the way to ever-increasing complexity. Thus, searching for tile sets capable of open-ended evolution, we first see that a tile set plays the role of a ''chemistry'' in the sense that it it defines the rules by which tile-based ''organisms'' can grow and function, and we further expect that we are looking for a tile set that displays some sort of universality of behavior.. Following this intuition, we argue that tile-based crystals can exploit computation to enhance their growth rate, and that this can lead to evolutionary processes resulting in increasing complexity of crystal structure. We introduce a framework for studying the evolution of metabolic control in crystals under resource-limited growth conditions. Within this framework, we design tile sets and environments that give rise to evolutionary landscapes in which crystals that perform more effective computations are fitter. We provide two main examples. First, in order to provide simple examples of metabolic evolution, we describe a tile set that can encode programmable logical computations. Second, to emphasize that arbitrarily complex computations can in principle be used by crystals to sense and respond to their environment, we exhibit a tile set capable of universal computation.. 2 Zig-zag ribbon evolution Previously, it was suggested that DNA tile assemblies could in principle evolve through cycles of crystal growth and splitting (Schulman and Winfree 2005) (Fig. 1) . The zig-zag tile set described in that work produces ribbon-like crystals that copy information along the length of the crystal. The tile set includes a group of square tiles, and two rectangular tiles called double tiles. Logical representations of the tiles that comprise a basic zig-zag ribbon are shown in Fig. 2a .. When growth occurs according to the tile assembly model, tiles are added to a ribbon in a zig-zag pattern shown in Fig. 2b . Only tiles that match at least two edges can attach. Given this constraint, the design of the tiles is such that at any moment there is just one tile that may be added to each end of the ribbon. The addition of each new row can be viewed as the copying of the information in the previous row. Using the tile set shown in Fig. 2a , this copying is trivial-only one sequence type is possible. However, the requirement that a tile attach by two bonds means that it must match both its vertical neighbor, either above and below, and its horizontal neighbor, to the left or right. In the case that several tile types may match the vertical neighbor, as shown in Fig. 2c , only the tile type that already appears in the row will match the horizontal neighbor. Only this tile can be added, so that information is inherited from layer to layer. As an example, the tile set shown in Fig. 2c has two tiles for each position and therefore can propagate one of four strings. This construction can be generalized to an additional number of bits by adding tiles to the tile set in Fig. 2c-2 n sequences can be propagated by a tile set containing 4n + 2 tiles. Further, a related tile set containing only six tiles can copy binary sequences of arbitrary width.. The growth of a crystal increases the number of copies of the original information present in the ribbon but does not produce any new growth fronts, so copying occurs at a constant rate. This information copying can be accelerated by periodic forces that cause ribbons to break. For each new ribbon that is created by breakage, two new growth fronts become available. Repeated fragmentation will therefore exponentially amplify an initial piece of information. Occasionally, a tile matching only one bond rather than two will join the assembly, resulting in a copying error, which will also be inherited. Such copying errors, inevitable in any physical implementation of tiles (e.g., Winfree 1998), will lead to evolution if ribbons with certain sequences grow faster than others. While it is possible to increase the complexity of selective pressures by increasing the complexity of tile sets used for growth, we've suggested here that tile sets exist that allow crystals to encode and run programs that can predict arbitrarily complex changes in the environment. Evolution using such a tile set as a medium could produce very complex sequences that grow well because they are particularly good at predicting changes in growth conditions. This suggests that crystal growth chemistries logically can support open-ended evolution with arbitrarily complex fitness landscapes. Our argument that some assemblies will be more fit relies only on their growth rates, which are determined by how effective the assemblies are at predicting the availability of resource tiles. However, as Sect. 3 illustrates, to be fit, assemblies must not only grow quickly, but also shear frequently. Not enough is known about shearing frequencies to make confident predictions, as these rates are surely dependent on the kind of forces that produce shearing in practice. However, it seems safe to assume that wider zig-zag assemblies would shear less frequently than thin ones in most cases. If this were the case, of two assemblies that predicted the availability of resource tiles equally well, the thinner Fig. 5 shows how to translate an arbitrary Turing machine into a tile set that simulates it. Applying this construction to a universal Turing machine (UTM) yields a tile set that forms zig-zag ribbons of the type shown here: the program being executed by the UTM is stored in the ribbon separate from the work tape used by the program. (b) A tile set that simulates a universal Turing machine can be combined with boundary tiles that are either resource or measurement tiles. Tiles can also be added to load the input from measurement tiles onto the work tape and to output the desired binding site for the resource tile. With such a tile set, crystals that simulate a Turing machine that correctly predicts the concentrations of resource tiles over time would be most fit assembly would be more fit. Such an effect could actually have the effect of favoring correct and concise programs for prediction of resource tile concentrations, i.e., it is a natural implementation of Occam's razor. This scenario is somewhat akin to Solomonoff inference (Solomonoff 1964) and Levin search (Levin 1973) in that short programs that produce correct results are found by biased random search. However, our arguments in this paper are limited: they neglect several important real life features of crystal growth. For example, while the model used in this paper prohibits any tile that matches fewer than two bonds from attaching, such attachments occur at a rate dependent on the physical conditions of assembly. If this error rate is too large, two things can happen. First, particularly large programs that take a long time to run may have difficulty accurately computing without mistakes, and there may be a preference for socalled robust programs (if such programs exist for the tile set used), which compute an answer correctly even with a couple of assembly errors. Second, it may be possible for assemblies that ask for the absent resource tile to eventually attach the available but nonmatching resource tile. Also, we have also not considered the effects of backward growth on the fitness landscape of crystals. For some tile sets, backward growth is not deterministic and quickly leads to a configuration that cannot be continued except by making an error, in which case backward growth stalls and can be neglected. For other tile sets, backward growth is deterministic and produces the same class of patterns that can be produced by forward growth. Yet another simplification we have made is to ignore stochastic effects during assembly and how they affect the time at which an assembly asks for a resource tile. Again, we do not expect such considerations do significantly change our general conclusions.. Despite these and other limitations to our study as presented here, we believe that the qualitative features of the mechanisms that we describe should be preserved in a more realistic model of crystal growth. While the tile sets we described here are too complex to implement experimentally at this time, it is not unreasonable to believe that much simpler tile sets could produce complex adaptations in response to resource limitations. Our initial investigations into whether the ideas here could be tested in experiments suggest that tile sets containing as few as ten tile types could in fact exhibit complex evolution in an environment with constant tile concentrations, as will be described elsewhere. Such a small tile set and environment would be amenable to laboratory experiments to test whether nontrivial crystal genotypes that efficiently use available resources could evolve. While selection for larger and larger genotypes require more and more accurate copying of information (Eigen 1971) and selection based on smaller fitness differences, error-resistant tile sets (Winfree and Bekbolatov 2004; Chen and Goel 2004; Reif et al. 2004) , which contain more tiles but copy more accurately could be used. In principle these tile sets can reduce error rates as much as is desired. One might think that for the fittest species in a given environment to be arbitrarily complex, the environment must be equally complex. But it is not immediately clear whether this is actually the case. Since there is no fastest algorithm for some functions (Blum 1967) , this would seem to imply that in a crystal growth environment that requires crystals to compute such a function, evolutionary pressure favoring speed would lead to ever-increasing complexity. On other hand, faster programs are in general larger. So while it is possible that evolutionary pressure might favor faster algorithms, the crystal must also be wider in order to contain the complex program and copying the program slows down the crystal's growth. This trade-off is also present in biological, in vitro, and artificial evolution.. It is also interesting to ask about the minimal requirements for open-ended evolution of crystals. Is it possible that there is a ''universal'' environment where tile concentrations are constant or vary in a simple periodic pattern, but open-ended evolution is still possible? To answer such a question, it may be worth considering effects that are beyond the simple model of crystal growth and replication kinetics used in this paper. We considered only fitness of species for growth in the exponential phase, where resources are not depleted by other crystals, and where there is no interaction between crystals. Relaxing these assumptions may allow other interesting routes to complexity. For example, could competition between crystals for tiles lead to an ''arms-race'' of more and more complex strategies for growth? Conversely, might symbiosis between crystal programs lead to interesting phenotypes not explored here? Might a crystal growth chemistry support parasitic crystals that evolve to grow by using existing crystals rather than single tiles? Supporting this last possibility, initial experiments with DNA crystals indicate that joining of crystals does occur (Ekani-Nkodo et al. 2004 ).

O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O B O O B O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O B O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O B O O O O O O O O B O O O O B O O O O O O O


The Next 700 BFT Protocols In this section, we present our new approach for the design and reconfiguration of abortable state machine replication protocols. We start with an overview of Abstract, the new abstraction we propose. Then, we illustrate the way Abstract can be used to design BFT protocols. Finally, we provide the formal specification of Abstract and state and prove the Abstract composability theorem. We present Abstract (ABortable STate mAChine replicaTion), a new abstraction for designing and reconfiguring generalized replicated state machines that are, unlike traditional state machines, allowed to abort executing a client's request if "something goes wrong."Abstract can be used to considerably simplify the incremental development of efficient Byzantine faulttolerant state machine replication (BFT) protocols that are notorious for being difficult to develop. In short, we treat a BFT protocol as a composition of Abstract instances. Each instance is developed and analyzed independently and optimized for specific system conditions. We illustrate the power of Abstract through several interesting examples.We first show how Abstract can yield benefits of a state-of-the-art BFT protocol in a less painful and errorprone manner. Namely, we develop AZyzzyva, a new protocol that mimics the celebrated best-case behavior of Zyzzyva using less than 35% of the Zyzzyva code. To cover worst-case situations, our abstraction enables one to use in AZyzzyva any existing BFT protocol.We then present Aliph, a new BFT protocol that outperforms previous BFT protocols in terms of both latency (by up to 360%) and throughput (by up to 30%). Finally, we present R-Aliph, an implementation of Aliph that is robust, that is, whose performance degrades gracefully in the presence of Byzantine replicas and Byzantine clients. State machine replication (SMR) is a software technique for tolerating failures using commodity hardware. The critical service to be made fault tolerant is modeled by a state machine. Several, possibly different, copies of the state machine are then placed Authors' addresses: P.-L. Aublin, INSA Lyon, 20 Avenue Albert Einstein, F-69621 Villeurbanne, France; email: pierre-louis.aublin@liris.cnrs.fr; R. Guerraoui, EPFL, LPD (Station 14), CH-1015 Lausanne, Switzerland; email: rachid.guerraoui@epfl.ch; N. Knežević and M. Vukolić (Current address), IBM ResearchZurich, Säumerstrasse 4, CH-8803 Rüschlikon, Switzerland; emails: {kne, mvu}@zurich.ibm.com; V. Quéma, Grenoble INP / ENSIMAG, 220, rue de la chimie, F-38400 Saint-Martin d'Hères, France; email: vivien.quema@imag.fr. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies show this notice on the first page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 2 Penn Plaza, Suite 701, New York, NY 10121-0701 USA, fax +1 (212) 869-0481, or permissions@acm.org. higher throughput than state-of-the-art protocols. Aliph uses, besides the Backup instance used in AZyzzyva (to handle the cases with asynchrony/failures), two new instances: (i) Quorum, targeted for system conditions that do not involve asynchrony/failures/contention, and (ii) Chain, targeted for high-contention conditions without asynchrony/failures. Quorum has a very low latency (like, e.g., Brasileiro et al. [2001] , Abd-El-Malek et al. [2005] , and Dobre and Suri 2006] ), and it makes Aliph the first BFT protocol to achieve a latency of only two message delays with as few as 3 f + 1 servers. Chain implements a pipeline message pattern and relies on a novel authentication technique. It makes Aliph the first BFT protocol with a number of MAC operations at the bottleneck server that tends to 1 in the absence of asynchrony/failures. This contradicts the claim that the lower bound is 2 [Kotla et al. 2010] . Interestingly, each of Quorum and Chain could be developed independently and required less than 35% of the code needed to develop state-of-the-art BFT protocols. 4 (3) R-Aliph, a protocol based on Aliph that achieves about the same performance as Aliph in the absence of faults, and that performs significantly better than Aliph under attack, that is, when Byzantine replicas and Byzantine clients act maliciously in order to decrease the performance of the system. Similarly to Aliph, R-Aliph uses the Quorum and Chain protocols when there are no attacks. In order to achieve good performance under attack, R-Aliph monitors the progress of Chain and Quorum, implements various mechanisms to bound the time required to switch from one protocol to another protocol, and uses the Aardvark protocol [Clement et al. 2009] as Backup instance. Aardvark is a BFT protocol that has been specifically designed to sustain good performance despite attacks.. The rest of the article is organized as follows. Section 2 describes the system model. Section 3 presents Abstract. Afterward, we describe our new BFT protocols: AZyzzyva in Section 4, Aliph in Section 5, and R-Aliph in Section 6. Section 7 discusses the related work. Finally, Section 8 discusses future work and concludes the article. Correctness arguments and proofs of our implementations are postponed to Appendix A. Abstract has been devised with Byzantine faults in mind, but it is not restricted to them. In fact, Abstract specification does not explicitly refer to any specific kind of faults.. Individually, each Abstract instance behaves just like a (replicated) state machine: it commits clients' requests, returning state-machine-dependent replies to clients. Each reply is a function of a sequence of clients' requests called a commit history [van Renesse and Guerraoui 2010] . As in traditional state machine replication [Schneider 1990 ], commit histories are totally ordered-we refer to this property of Abstract as Commit Order. However, an Abstract instance needs to commit a request only if certain Progress conditions are met; otherwise, an Abstract instance may abort a client's request. These Progress conditions, determined by the developer of the particular Abstract instance, might depend on the design goals and the environment in which a particular instance is to be deployed. Intuitively, designing a variant of a replicated state machine, in particular a Byzantine fault-tolerant one, that needs to make progress only when certain conditions are met is often considerably simpler than designing a full-fledged replicated state machine that makes progress under all system conditions.. A single Abstract instance is not particularly interesting on its own. That is why the Abstract framework comes with a mechanism and formalism for reconfiguring Abstract instances. Abstract reconfiguration resembles traditional state machine reconfiguration [Lamport et al. 2010; Birman et al. 2010] , with the difference that its properties are tailored for reconfiguration of abortable state machines rather than full-fledged, traditional ones. In a sense, Abstract provides a rigorous framework of reconfiguration as an object, using shared-memory style.. More specifically, for the sake of reconfiguration, every Abstract instance has a unique identifier (called instance number) i; this instance number abstracts away a set (and a number) of replicas implementing a given instance, the protocol used, possible protocol internals such as a protocol leader, and so forth. For example, in the BFT protocols that we present in this article, Abstract instance number i captures the protocol used and its Progress conditions. Following the classical state machine reconfiguration approach [Lamport et al. 2010; Birman et al. 2010] , Abstract reconfiguration can be divided into three steps: (i) stopping the current Abstract instance, (ii) choosing the next Abstract instance, and (iii) combining the request sequences (i.e., commit histories) of separate Abstract instances into a single sequence.. (i) An Abstract instance is automatically stopped as soon as it aborts a single request. This is captured by the Abstract Abort Order property. Namely, when a client's request aborts, along the abort indication, the Abstract instance also returns a sequence of requests called abort history. The Abort Order property mandates that each abort history of a given Abstract instance i contains as its prefix every commit history of instance i (possibly along with some uncommitted requests). In a sense, Abstract Abort Order allows a certain request index to become a stopping index-any attempt to order requests higher than this index is guaranteed to abort, effectively causing a given Abstract instance to "stop." As we will see later on, an abort history is used as an input to the next Abstract instance. (ii) Within an abort indication, an aborting Abstract instance i also returns an identifier of the next instance next(i)-we say instance i switches to instance next (i) .. As in the reconfiguration of classical state machines [Lamport et al. 2010] , the Abstract switching requires consensus on the next Abstract instance; hence, we require next to be a function, that is, to have next(i) denote the same instance across all abort indications of instance i. Moreover, since we stop aborting instances and to avoid "switching loops," we require monotonically increasing instance numbers, that is, for every instance i, next(i) > i.. In the context of the protocols presented in this article, we consider next(i) to be a predetermined function (e.g., known to servers implementing a given Abstract instance); we talk about deterministic or static switching. Concretely, in our protocols, we simply fix next(i) = i +1. However, this is not required by our specification; next(i) can be computed "on the fly" by an Abstract implementation (e.g., depending on the current workload or possible failures or asynchrony). In this case, we talk about dynamic switching; this is out of the scope of this article. (iii) Finally, "state transfer," that is, migration of histories between Abstract instances, reveals critical Abstract subtleties. In short, the client uses the abort history h of an aborting Abstract instance i to invoke next(i); in the context of next(i), h is called an init history. Roughly speaking, init histories are used to initialize instance next(i), before next(i) starts committing/aborting clients' requests. However, Abstract mandates no explicit agreement on the initial state (i.e., initial sequence of requests) of the next instance next(i). More specifically, there is no agreement across abort histories of an aborting instance i. Abort histories need only contain commit histories as their prefix-there is no mutual order imposed on any two abort histories, which are free to contain arbitrary uncommitted requests at their tails.. Similarly, in the context of next(i), no specific init history (recall that init history of instance next(i) is an abort history of instance i) needs to be a prefix of commit/abort histories of next(i). Abstract only requires implicit agreement on the initial sequence of requests of next(i). This is captured by the Abstract Init Order property, which mandates that, for any Abstract instance, a longest common prefix of init histories is a prefix of any commit/abort history. Intuitively, since every commit history of an aborting instance i is contained as a prefix in every abort history of i, we do not need a stronger property than Init Order to maintain total order of commit histories across Abstract instances.. Such weak, implicit ordering of abort/init histories enables very efficient Abstract implementations. Since no explicit agreement is required neither across abort histories nor on the "first" init history, Abstract implementations do not need to solve consensus outside Progress conditions. In fact, and as our Abstract implementations (ZLight, Quorum, and Chain) exemplify, some Abstract specifications with "weak" Progress can be implemented in the asynchronous model despite Byzantine faults, circumventing the FLP impossibility result [Fischer et al. 1985 ].. With such a design, our Abstract framework has the following appealing properties that simplify the modular and incremental design of state machine replication, and BFT protocols in particular:. (1) Switching between instances is idempotent: the composition of two Abstract instances yields another Abstract instance. (2) A correct implementation of an Abstract instance always preserves the safety of a state machine, that is, the total order across committed requests [Schneider 1990; van Renesse and Guerraoui 2010] . This extends to any composition of Abstract instances. (3) A (replicated) state machine is nothing but a special Abstract instance-one that never aborts.. Consequently, the designer of a state machine replication (e.g., BFT) protocol has only to make sure that (1) individual Abstract implementations are correct, irrespectively of each other, and (2) the composition of the chosen instances is live: that is, that every request will eventually be committed. Byzantine fault-tolerant state machine replication (BFT) protocols are notoriously difficult to design, implement, and prove correct. In this article, we presented Abstract, a framework for the design and reconfiguration of abortable replicated state machines. Using Abstract, we incrementally developed new BFT protocols with a fraction of the complexity required to develop full-fledged BFT protocols. We build BFT protocols as sequences of Abstract instances, each designed, implemented, and proved independently. Such protocols are not only simpler to design but also efficient and robust.. In this article, we have implemented several BFT protocols (AZyzzyva, Aliph, and R-Aliph) that consist of lightweight Abstract implementations (ZLight, Quorum, and Chain) designed to be efficient in the "common case," typically when the system is synchronous and there are no replica faults. In all protocols, we reused existing BFT protocols such as PBFT [Castro and Liskov 2002] and Aardvark [Clement et al. 2009 ] to handle the faulty case by wrapping them into a powerful Abstract instance called Backup.. In future work, several directions can be interesting to explore, for example, using the concepts that underly Abstract in the context of Byzantine-resilient storage [Hendricks et al. 2007] or devising signature-free switching. Moreover, we believe that an interesting research challenge lies in devising effective heuristics for dynamic switching among Abstract instances. While we described Aliph and R-Aliph and showed that, albeit simple, they outperform existing BFT protocols, Aliph and R-Aliph are simply the starting point for Abstract. The idea of dynamic switching depending on the system conditions seems very promising; such a scheme could monitor the current system conditions and implement smart heuristics to switch to the seemingly most appropriate Abstract instance. The idea of aborting if "something goes wrong" is old. It underlies, for instance, the seminal two-phase commit protocol [Gray 1978 ]: abort can be decided if there is a failure or some database server votes "no." The idea was also explored in the context of mutual exclusion: a process in the entry section can abort if it cannot enter the critical section [Jayanti 2003 ]. Abortable consensus was proposed in Chen [2007] and Boichat et al. [2003] . In the first case, a process can abort if a majority of processes cannot be reached, whereas in the second case, a process can abort if there is contention. The latter idea was generalized for arbitrary shared objects in Attiya et al. [2005] and then Aguilera et al. [2007] . In Aguilera et al. [2007] , a process can abort and then query the object to seek whether the last query of the process was performed. This query can, however, abort if there is contention. Our notion of abortable state machine replication is different. First, the condition under which Abstract can abort is a generic parameter: it can express, for instance, contention, synchrony, or failures. Second, in case of abort, Abstract returns (without any further query) what is needed for recovery in a Byzantine context; namely, an unforgeable history. This, in turn, can be used to invoke another, possibly stronger Abstract. This ability is key to the composability of Abstract instances.. Several examples of speculative protocols, distinguishing an optimistic phase from a recovery one, were discussed in the survey of Pedone [2001] . These speculation ideas were used in the context of Byzantine state machine replication, for example, in HQ [Cowling et al. 2006] and Zyzzyva [Kotla et al. 2010] . We are, however, the first to clearly separate the phases and encapsulate them within first-class, well-specified modules that can each be designed, tested, and proved independently. In a sense, Abstract enables the construction of a BFT protocol as the composition of as many (gracefully degrading) phases as desired, each with a "standard" interface. This allows for an unprecedented flexibility in BFT protocol design that we illustrated with Aliph, a BFT protocol that combines three different phases. Similarly, with R-Aliph, we illustrated how one can quickly devise and implement a BFT protocol ensuring good performance despite attacks. While we described Aliph and R-Aliph and showed that, albeit simple, they outperform existing BFT protocols, Aliph and R-Aliph are simply the starting point for Abstract.. To maintain the assumption of a threshold f of replica failures, BFT systems need to ensure failure independence [Gashi et al. 2007; Garcia et al. 2011 ]. An established technique used in ensuring failure independence is n-version programming, which mandates a different BFT implementation for each replica, with the goal of reducing the probability of identical software faults across replicas. While Abstract does not alleviate the need for n-version programming, this may be less costly and more feasible due to the inherently reduced code sizes and complexities involved with Abstract implementations. In addition, abstractions like BASE [Castro et al. 2003 ], which enable reuse of off-theshelf service implementations, can be used complementarily to our approach.. Since the publication of the preliminary, conference version of this work , several papers that exploit Abstract-like reconfiguration have been published. In particular, CheapBFT [Kapitza et al. 2012] implements Abstract-like reconfiguration with switching through replicas assuming an FPGA-based trusted subsystem to reduce the resource overhead of BFT. In the common case, CheapBFT relies on a novel, optimistic protocol that uses f + 1 replicas, whereas in the case of failures, CheapBFT falls back to a protocol called MinBFT [Veronese et al. 2013] (itself also based on a trusted subsystem) that uses 2 f + 1 replicas. It is precisely for such usages demonstrated by CheapBFT, which diverge from the classical BFT model we assume in this article, that we defined Abstract as a specification with properties that reason about clients' histories rather than about the state and the number of replicas that implement Abstract. Indeed, Abstract specification deliberately does not reason about the number of replicas, their local state or state relative to other replicas, or the fault model. By adopting such an approach, we do not restrict the use of Abstract to the classical BFT model, but also allow for the use of Abstract in the trusted BFT model [Kapitza et al. 2012] , vanilla crash-failure model, or any other failure model.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


On the Equivalence Between Stein and De Bruijn Identities This paper focuses on illustrating 1) the equivalence between Stein's identity and De Bruijn's identity, and 2) two extensions of De Bruijn's identity. First, it is shown that Stein's identity is equivalent to De Bruijn's identity under additive noise channels with specific conditions. Second, for arbitrary but fixed input and noise distributions under additive noise channels, the first derivative of the differential entropy is expressed by a function of the posterior mean, and the second derivative of the differential entropy is expressed in terms of a function of Fisher information. Several applications over a number of fields, such as signal processing and information theory, are presented to support the usefulness of the developed results in this paper.Index Terms-Bayesian Cramér-Rao lower bound (BCRLB), Costa's EPI, Cramér-Rao lower bound (CRLB), De Bruijn's identity, entropy power inequality (EPI), Fisher information inequality (FII), Stein's identity. This paper mainly disclosed three information-estimation relationships. First, the equivalence between Stein identity and De Bruijn identity was proved. Second, it was proved that the first derivative of the differential entropy with respect to the parameter can be expressed in terms of the posterior mean. Third, this paper showed that the second derivative of the differential entropy with respect to the parameter can be expressed in terms of the Fisher information. Finally, several applications based on the three main results listed previously were provided. The suggested applications illustrate that the proposed results are useful not only in information theory but also in the estimation theory field and other fields.

O O O O O O O O O O O O O O O O O O B I O B I I O O O O O B I I O O O O O B I O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I I B I I I B I I I B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Evolutionary Rule Mining in Time Series Databases Abstract. Data mining in the form of rule discovery is a growing field of investigation. A recent addition to this field is the use of evolutionary algorithms in the mining process. While this has been used extensively in the traditional mining of relational databases, it has hardly, if at all, been used in mining sequences and time series. In this paper we describe our method for evolutionary sequence mining, using a specialized piece of hardware for rule evaluation, and show how the method can be applied to several different mining tasks, such as supervised sequence prediction, unsupervised mining of interesting rules, discovering connections between separate time series, and investigating tradeoffs between contradictory objectives by using multiobjective evolution. As data are available in ever increasing amounts, there is a need for automated knowledge discovery. One recent development is the use of evolutionary methods to create flexible and powerful mining solutions. Methods for knowledge discovery in sequence data are well developed, but this is one domain where evolution has not been thoroughly examined yet. In our work we use specialized hardware for sequence searching to enable the evolution of rules from sequence data, or, more specifically, time series data. In this study we have used various real-world time series data sets as well as random time series data as a baseline. We attempt to make predictions about whether the time series will go up or down (in the cases where this is feasible) and look for regularities in the data that might be useful to a human expert. Although methods exist for doing this sort of thing (see Section 1.1), our method has two main contributions: (1) it is highly flexible and configurable, in that one can decide which rule format and objective measure to use, and (2) the resulting models and rules are (at least in principle) human-readable.. The foundation for this work has been laid in several previous publications (Hetland & Saetrom, 2002 . In this paper we have collected our main results, compared them with other machine learning methods (Section 3.1) and extending them by using our method for finding relations between several time series (Section 3.2.4).. We do not contend that our method is better at time series prediction or classification than existing methods, although we do show that it is quite robust and has considerable predictive power. Instead, our main contribution lies in the flexibility of our method and the freedom it affords the data miner, in that he or she can specify a rule format and quality function suited for the application at hand. Also, the basic components of our method (that is, the discretization method and genetic programming) are well known; it is the application of these methods to the domain of sequence rule mining that is novel.. Our method is based on the availability of special-purpose pattern matching hardware (Halaas et al., 2004) . While this may be seen as a limitation, the new functionality that this hardware offers has, in fact, been one of the driving forces behind our work. In the interest of wider applicability, we have in the past done some work on using more readily available software-based retrieval techniques with our method . That line of inquiry is not pursued in this paper. Previous attempts at solving the problem of mining predictive rules from time series can loosely be partitioned into two types. In the first type, supervised methods, the rule target is known and used as an input to the mining algorithm. Typically, this can be specific events in (or possibly extrinsic to), the time series. Thus the goal is to generate rules for predicting these events based on the data available before the event occurred. The papers (Hetland & Saetrom, 2002; Weiss & Hirsh, 1998; Zemke, 1998; Povinelli, 2000) fall in this category. All of these use some form of evolutionary computation; (Hetland & Saetrom, 2002) uses genetic programming, while the others use genetic algorithms.. In the second type, unsupervised methods, the only input to the rule mining algorithm is the time series itself. The goal is to automatically extract informative rules from the series. In most cases this means that the rules should have some level of preciseness, be representative of the data, easy to interpret, and interesting (that is, novel, surprising, useful, and so on), to a human expert (Freitas, 2002) . This is the approach we take in this paper.. Of the existing attempts to tackle this problem, many rely on scanning the data and counting the occurrence of every legal antecedent and consequent (for example, Mannila, Toivonen, & Verkamo, 1997; Agrawal & Srikant, 1995; Höppner & Klawonn, 2001) . The rules are then ranked according to some measure of interestingness. This approach does, however, place some limitations on the rule format in order to make the task of counting all occurrences feasible. Others have focused on specific mining problems, such as detecting unusual movements (Martin & Yohai, 2001) , or finding unusual temporal patterns in market basket data (Chakrabarti, Sarawagi, & Dom, 1998) .. Unlike these approaches, we try to tackle the core problem directly, that is, mining interesting rules. This is done by defining some formal interestingness measure and using genetic programming to search the rule space for the most interesting rules. Thus, unlike other methods, the interestingness measure is used directly in the mining process and not as a post-processing ranking function. This allows for a much more flexible rule format than the existing methods.. Others have tried this direct approach in the context of standard database mining. For instance, Noda, Freitas, and Lopes (1999) used a genetic algorithm to find interesting association rules in two different relational databases. The algorithm evolved the antecedent and then selected the most accurate consequent from a pre-specified set of consequents. This approach is, however, new to time series mining. This section outlines the specifics of the problem we are trying to solve, and describes the variations of our data mining method in more detail. In our experiments we use four data sets from the UCR Time Series Data Mining Archive (Keogh & Folias, 2002) : ECG measurements from several subjects, concatenated; earthquake-related seismic data; monthly mean sunspot numbers from 1749 until 1990; and network traffic as measured by packet round trip time delays. Figure 2 shows plots of parts of the different time series analyzed.. In addition, in our unsupervised mining experiments we used stock series available from Yahoo finance.. 1 The series were the adjusted daily closing price of ten U.S. companies listed on the New York Stock Exchange. More specifically, the companies were Amgen, Dell, Disney, General Electric, General Motors, IBM, Microsoft, Oracle, Sun, and Walmart.

O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


A DSL to declare aspect execution order Composing aspects is known to be problematic since unpredicted aspect interactions may appear and may lead to erroneous weaved programs. This paper focuses on one of these issues: the advice ordering around a join point. It views aspect composition issues as a crosscutting concern that should be handled by a composition aspect. It proposes a domain-specific declarative aspect composition language for composing aspects, and applies it on a number of examples. The goal of aspect-oriented programming is separation of concerns, where each concern could be developed by an expert in the problem domain tackled by that concern. A domain specialist in transactional programming should be able to design a transactional aspect language that could then be composed in programs that need transactions. Initially these aspect languages were supposed to be domain-specific languages, as illustrated by for example the COOL language. Because of the complexity of designing weavers for more-or-less general purposes aspect languages, research on domain-specific aspect languages was taking the low road. Recently, as shown also by this workshop series, domainspecific aspect languages are on the rise again, for all kinds of domains.. This paper is concerned with the domain of aspect-oriented programming itself. It is a well-known AOP problem that composing multiple aspects, even if each aspect works correctly when being composed in isolation, is hard. We introduce a model where composition issues are considered as a crosscutting concern handled by a composition aspect implemented using our domain-specific aspect composition language. Our aspect composition language is declarative (to let composers focus on the actual composition) and allows to change composition orders at runtime (since this is needed in practice, especially when the composed aspects were developed in isolation).. Note that we submitted a paper similar to this one to the AOSD'08 workshop "6th Workshop on Software-engineering Properties of Languages and Aspect Technologies". We will attend both workshops with a very similar position on aspect composition, but with a different focus. This paper focuses on the domainspecificness of our aspect language. The other submission focuses on the non-functional properties (the '-ilities') of the language.. The rest of the paper is structured as follows. Section 2 looks in more detail at the problem of ordering advices. Section 3 describes related work, while Section 4 introduces our own solution. Section 5 gives examples to demonstrate how frequently occurring composition problems are addressed by the language. Section 6 discusses future work for this research. Section 7 concludes the paper. First of all we would like to finish our implementation using the extensible ASPECTBENCH COMPILER [1] and then perform a number of case studies on real examples. Last but not least we want to help composers of aspects spot composition problems and understand the semantics of the composition they are making. Composing aspects is not a trivial task. We view this problem as a cross-cutting concern in need of its domain-specific language that enables composers to express compositions easily. We propose a declarative aspect composition language based on only three concepts (composition rules, constraints and contexts) that directly deal with composition.. Composing aspects is known to be problematic since unpredicted aspect interactions may appear and may lead to erroneous weaved programs. This paper focuses on one of these issues: the advice ordering around a join point. It views aspect composition issues as a crosscutting concern that should be handled by a composition aspect. It proposes a domain-specific declarative aspect composition language for composing aspects, and applies it on a number of examples. The problem of ordering multiple aspects is not a new one, and there is a number of approaches and techniques that address this problem. This section reviews three techniques: ordering aspects, JAsCo, and declarative aspect composition.. Ordering aspects. One way to order advices is to let programmers express the order of aspects. If an aspect A is set to precede an aspect B then all advices of A will always be executed before any advice of B. This is the solution implemented in ASPECTJ [4] by the declare precedence statement. That technique is also used in [2] by the use of several operators between aspects. The major problem with this approach is that it makes it hard to specify advice-specific ordering (needed in our example). One could create an aspect for each advice that needs its order, but that solution modifies existing aspects and is awkward.. JAsCo. JAsCo [7] is an aspect-oriented extension of the Java language that combines AspectJ's expressive pointcuts with the Aspectual Components [5] approach of aspect independency. In JAsCo, the scope of an aspect is separated from its behavior. The crosscutting behavior is described in so-called hooks. A hook is composed of several advices and an abstract pointcut. Connectors are used to instantiate hooks and bind their abstract pointcuts to concrete ones in the program. Developers have to specify the advice sequence they want in the connector. Moreover hook composition strategies can be expressed that have the possibility to ignore hooks in particular composition contexts.. To implement our example using JAsCo, we would need at least four hooks, three connectors and one connector composition strategy (which is different from hook composition strategies). The connector composition strategies allows us to change dynamically the applying order of the connectors and thus of the hooks they contains. We need this since the advice ordering has to be altered if the file is sent by an older client version.. While JAsCo offers lots of flexibility it does so in a very lowlevel way. We expect a domain-specific approach to talk about aspect composition to be more high-level.. Declarative aspect composition A third approach is declarative aspect composition [6] , based on declaring constraints between advices. The system then computes a possible ordering that respects all of these constraints, if possible.. There are two kinds of constraints that can be declared: ordering constraints and control constraints. An ordering constraint declares that one advice has to be executed before another one. A control constraint declares that an advice will be executed only if another one succeeded. Execution orders cannot be changed dynamically, meaning that in our example we cannot implement the backward compatibility with older versions that use compressed encrypted files instead of the newer encrypted compressed files. What we like is that a composer can focus on the actual intended composition.

O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Fuzzy Color Histogram-based Video Segmentation We present a fuzzy color histogram-based shot-boundary detection algorithm specialized for contentbased copy detection applications. The proposed method aims to detect both cuts and gradual transitions (fade, dissolve) effectively in videos where heavy transformations (such as cam-cording, insertions of patterns, strong re-encoding) occur. Along with the color histogram generated with the fuzzy linking method on L*a*b* color space, the system extracts a mask for still regions and the window of picture-in-picture transformation for each detected shot, which will be useful in a content-based copy detection system. Experimental results show that our method effectively detects shot boundaries and reduces false alarms as compared to the state-of-the-art shot-boundary detection algorithms. Recent developments in multimedia technology with the significant growth of media resources introduced content-based copy detection (CBCD) as a new research field alternative to the watermarking approach for identification of video sequences. The detection of shots, as in many video indexing and retrieval applications, is the first step of video analysis. A shot is defined as a series of related consecutive frames representing a continuous action in time and space taken by a single camera [1] .. A video is composed of several shots combined with abrupt or gradual transitions. An abrupt transition, also known as hard-cut, is the most common and easy to detect transition type. On the other hand, gradual transitions (fades, dissolves and wipes) are spread over a number of frames, thus they are harder to detect.. Various shot-boundary detection algorithms have been proposed [1] [2] [3] [4] [5] [6] and compared [7] [8] [9] [10] ; however, to the best of our knowledge, no shot-boundary detection algorithm specialized for CBCD is found in the literature. Our aim is to propose an automatic shot-boundary detection algorithm for the videos on which various transformations are applied. In contrast to most of the existing methods, we utilize fuzzy logic approach for extracting color histogram to detect shot boundaries. We evaluate the proposed method in the query dataset prepared for CBCD task of TRECVID 2008, and show the accuracy of the system. The paper is organized as follows: Section 2 discusses related work. Section 3 explains the video transformations used for preparing a dataset for CBCD. Section 4 describes the proposed shot-boundary detection algorithm, as well as the methods for detecting frame-dropping and picture-in-picture transformations, noise detection, and mask generation. We present experimental results and evaluate the performance of the proposed methods in Section 5. Section 6 gives conclusions and future work. We propose a fuzzy color histogram-based shot-boundary detection method for the videos where heavy transformations (such as cam-cording, insertions of patterns, strong re-encoding) occur. In addition to detecting shot-boundaries using fuzzy color histogram, we extract a mask for still regions and the window of picture-in-picture transformation. Experimental results show that the proposed method effectively detects shot boundaries with a small false alarm rate as compared to the state-of-the-art shotboundary detection algorithms.. As a future work we will use the detected shot boundaries, masks of still regions, picture-in-picture window boundaries, and the fuzzy color histogram method in our content-based copy detection system. Studies on shot-boundary detection are typically based on extracting visual features (color, edge, motion, and interest points) and comparing them among successive frames. Truong et al. [6] propose techniques for cut, fade, and dissolve detections. An adaptive thresholding technique to detect peaks in the color histogram difference curve is presented for detecting hard cuts. Locating monochrome frames and considering luminance mean and variance are the steps for fade and dissolve detection. Danisman and Alpkocak [11] apply a method based on color histogram differences in RGB color space and thresholding for cut detection. They present skip frame interval technique, which reduces the computation time with a slight decrease in the precision. Dailianas et al. [9] , Boreczky and Rowe [10] compare early shot-boundary detection algorithms. Lienhart [8] extends this comparison by taking newer algorithms into account, and by measuring their ability to detect the type and temporal extent of the transitions. Cotsaces et al. [7] give an up-to-date review.. In recent years, researchers focus on detecting gradual transitions effectively and avoiding the false alarms caused by flashlight and the motion of large objects in the scene, since the recognition of hard-cuts is very reliable for most of the methods. Huang et al. [2] propose an approach based on local keypoint matching of video frames to detect abrupt and gradual transitions. By matching the same objects and scenes using contrast context histogram (CCH) in two adjacent frames, the method decides that there is no shot change. Grana et al. [3] propose a two-step iterative algorithm, unique for both cuts and gradual transitions detection, in the presence of fast object motion and camera operations. Boccignone et al. [1] use a consistency measure of the fixation sequences generated by an ideal observer looking at the video for determining shot changes. A scene-break detection approach based on linear prediction model is proposed in [4] . Shot-boundaries are detected using Bayesian cost functions, by comparing original frame with the predicted frame, estimated using within video shot linear prediction model (WLPM) and dissolve linear prediction model (DLPM). Yuan et al. present a unified shot boundary detection system based on graph partitioning model [5] . The representation of the visual content, the construction of the continuity signal, and the classification of continuity values are handled in this work. The evaluations show that the SVM-based active learning outperforms both thresholding and nonactive learning.. Fuzzy logic introduced by Zadeh [12] is being used in many applications related to image processing. Konstantinidis et al. [13] and Han and Ma [14] utilize fuzzy logic for creating color histograms to be used in content-based image retrieval systems. Chung and Fung [15] introduce fuzzy color quantization to color histogram construction, and evaluate its performance in video scene detection with a very limited video dataset. Fang et al. [16] propose a fuzzy logic approach for temporal segmentation of videos, where color histogram intersection, motion compensation, texture change and edge variances are integrated for cut detection. In [17] , histogram differences of consecutive frames are characterized as fuzzy terms, such as small, significant and large, and fuzzy rules for detecting abrupt and gradual transitions are formulated in a fuzzy-logic-based framework for segmentation of video sequences. Das et al. [18] define a unified interval type-2 fuzzy rule based model using fuzzy histogram and fuzzy co-occurrence matrix to detect cuts and various types of gradual transitions.. In the field of CBCD, representing video with a set of keyframes (one or more representative frame for each shot) is a common approach. Some of the recent studies on CBCD task of TRECVID 2008 employ the following techniques. Llorente et al. [19] use an approach based on color histogram and thresholding, extended by [20] for detection of gradual transitions. Douze et al. prefer extracting 2.5 frames per second for query videos, and extracting only a few representative keyframes for the dataset [21] . We also preferred extracting a fixed number of frames per time interval in our CBCD system [22] . Studies in video copy detection domain, therefore, do not necessarily use a shot-boundary detection method. We provide the details of our shot-boundary detection method and other techniques that we use to identify the transformations applied on a query video. The parameters of the system are given in Table 2 , and the overview of the proposed algorithm is presented in Fig. 2 .

O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


A novel multi-view image coding scheme based on view-warping and 3D-DCT Efficient compression of multi-view images and videos is an open and interesting research issue that has been attracting the attention of both academic and industrial world during the last years. The considerable amount of information produced by multi-camera acquisition systems requires effective coding algorithms in order to reduce the transmitted data while granting good visual quality in the reconstructed sequence. The classical approach of multi-view coding is based on an extension of the H.264/AVC standard, still based on motion prediction techniques. In this paper we present a novel approach that tries to fully exploit the redundancy between different views of the same scene considering both texture and geometry information. The proposed scheme replaces the motion prediction stage with a 3D warping procedure based on depth information. After the warping step, a joint 3D-DCT encoding of all the warped views is provided, taking advantage of the strong correlation among them. Finally, the transformed coefficients are conveniently quantized and entropy coded. Occluded regions are also taken into account with ad-hoc interpolation and coding strategies. Experimental results performed with a preliminary version of the proposed approach show that at low bitrates it outperforms the H.264 MVC coding scheme on both real and synthetic datasets. Performance at high bitrates are also satisfactory provided that accurate depth information is available. In recent years, there has been a growing interest in three-dimensional representation, especially in 3D video systems. While commercially-available stereoscopic video systems allow the viewer to perceive the depth of the scene from a given viewpoint, an unconstrained browsing of the scene from an arbitrary viewpoint (Free-Viewpoint Video or FVV) is a much more complex issue on which research is still very active. Indeed if a depth map associated to a single view of the scene permits warping the view to a novel viewpoint, one needs several views and depth maps in order to achieve satisfactory viewing freedom and to avoid occlusions [1] .. The so-called multi-view plus depth representation permits to freely browse a three-dimensional scene represented by multiple views and depth maps as if a 3D model was available. In this way, all the burden connected to the construction of the 3D representation is avoided. This approach nowadays appears to be one of the most effective solutions to the challenges of real-time 3DTV applications. However, in order to permit a satisfactory free browsing of the scene, a large number of views has to be transmitted, and their compression is a fundamental step in the construction of a practical multi-view system.. Although conceivable, compressing multi-view representations independently by standard single-view image and video coding techniques (eventually with adhoc optimizations [2] ) is not very efficient since there is a quite relevant redundancy between different views of the same scene that a similar approach would not use. An effective coding scheme should be able to exploit such a redundancy in order to obtain an optimal compression of the image or video data. To this purpose several solutions have been proposed in the literature. Some of these schemes aim at reducing the inter-view redundancy by some low-pass transformation across the pixels of the different views, like in [3] . This work is based on a 3D-DCT, but the transform is directly applied to the source images without any disparity compensation method. As a result the encoding method of [3] can not fully exploit the energy compaction property of the DCT transform.. Other solutions apply motion compensation techniques, typically adopted to reduce temporal redundancy, in order to predict one view from another relative to a different viewpoint. Following this trend, the video coding standard H.264 MVC [4] , developed from the scalable video coder H.264 SVC [5] , obtains high compression gains by predicting each frame of a single view both from previous data and from the other views. A disparity based method for motion vectors' prediction is introduced in [6] for stereoscopic video compression. It represents an interesting and up-to-date possibility. 3D warping of the temporal-motion vectors in one view is used to predict the motion vectors in the other one. Warping is performed by an affine transformation based on a disparity model updated frame by frame.. Among the class of Multi-View Coding (MVC) schemes employing motion compensation, one must also mention multi-view video coders based on Distributed Video Coding (DVC) techniques [7] . In this case, each frame is coded according to the correlation existing among the different views, and motion compensation techniques are employed at the decoder. The proposed solutions allow reducing the computational complexity at the encoder and mitigating the effects of data losses on the reconstructed 3D sequences.. Despite the fact that these approaches make possible to reuse a huge amount of previous work on video compression, they do not fully exploit the three dimensional nature of multi-view data. For example, the location of corresponding samples in different views is given by the three dimensional structure of the scene (given by the depth maps), and it is possible to employ this knowledge instead of the standard block-based motion prediction [8] .. Moving further, it is possible to exploit different schemes than the classical ones based on motion prediction and residual coding. Even if in the literature it is possible to find some examples of video coders based on three dimensional transforms involving the spatial and temporal dimensions [9] , the use of this kind of representations is not very common because of the problems with scene changes and the unreliability of motion prediction. Another interesting possibility is given by the method of [10] which provides scalability and flexibility as well as good coding performances exploiting a multi-dimensional wavelet transform combined with an ad-hoc disparity-compensated view filter (DCVF).. This work introduces a novel lossy compression scheme for multi-view data which tries to fully exploit the inter-view redundancy in order to achieve better compression performance. A video coder is essentially made of a redundancy reduction mechanism followed by a coding part. The focus of this work is on the first element, i.e., on the exploitation of inter-view redundancy. The novel elements of the proposed scheme with respect to H.264 MVC or similar schemes are several. Firstly the inter-view motion prediction stage is replaced by a 3D warping procedure based on depth information. Then, the traditional 2D-DCT is replaced by a multi-dimensional transform, namely a 3D-DCT. However, in the proposed method, differently from [3] , the 3D-DCT is applied on disparity compensated data, in order to better exploit view-redundancy in the transform stage. The transform coefficients are then coded by standard entropy coding algorithms. Finally, a novel interpolation and compression scheme is introduced in order to handle the regions that are occluded or out of the viewing field in some views but not in others.. The rest of the paper is organized as follows. Section 2 presents the basic ideas that lie behind the adopted compression strategy. Section 3 provides an overview of the adopted coder describing its main parts. Experimental results in Section 4 show that the proposed approach can improve the average PSNR value over the different views up to 2 dB with respect to H.264 MVC encoding performed with comparable coding features. Finally, conclusions are drawn in Section 5. This paper introduces a novel coding scheme for multi-view images. Differently from most currently available solutions, the proposed approach is not an extension of any standard image or video compression tool, but it is instead based on an innovative algorithm explicitly targeted to multi-view data. Firstly 3D warping is used in place of the classical motion prediction stage in order to improve the motion compensation accuracy and to avoid the limitations given by fixed block sizes. Another original contribution is the use of a 3D-DCT transform in order to directly exploit inter-view redundancy in the transform stage instead of relying on the standard prediction and residual approach.. Finally, the critical issue of occluded areas has also been taken into account with an ad-hoc interpolation and coding strategy. Experimental results show how the proposed algorithm outperforms H.264 MVC at low bitrates in most practical configurations. However, the performance of the method at high bitrates depends on the accuracy of the available geometry information.. Further research will initially be focused on improving the performance of the transform stage as well as the entropy coding stage with the inclusion of features like adaptive block size, in-loop deblocking filtering, and R-D optimization that are still missing in the current implementation. Moreover, at high bitrates the coding gain can be utterly improved by optimizing the adopted context-adaptive binary arithmetic coder (i.e., modifying the current structure of contexts and the function that maps syntax elements into binary strings). Another important research direction aims at making the system more robust with respect to unreliable depth information. Optimized solutions for the compression of the depth data, possibly exploiting some of the techniques proposed in this paper, will also be included in the proposed framework. Another improvement will be the inclusion of a motion prediction stage in order to exploit this scheme also in the compression of multi-view video. Finally, scalability issues will also be considered with particular focus on view scalability. In order to achieve this target, considering also the very promising results of [10] , we will also take into account the possibility of replacing the 3D-DCT with more flexible transforms like the 3D/4D wavelet. Examples of occlusion detection when depth data of both reference view and target view are available. In the a) case, P coincides with P ′ so the scene's point is correctly viewed by both cameras. In the b) case, P and P ′ differs because the target camera is not able to see P. In this case, an occlusion is detected and the target pixel is not used in the warped image. . . . . . . . . . All the eight warped views in "breakdancers" dataset before (V i→4 , i = 0, . . . , 7) and after (V

O O B I I O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O