A random walk approach to sampling hidden databases A large part of the data on the World Wide Web is hidden behind form-like interfaces. These interfaces interact with a hidden backend database to provide answers to user queries. Generating a uniform random sample of this hidden database by using only the publicly available interface gives us access to the underlying data distribution. In this paper, we propose a random walk scheme over the query space provided by the interface to sample such databases. We discuss variants where the query space is visualized as a fixed and random ordering of attributes. We also propose techniques to further improve the sample quality by using a probabilistic rejection based approach. We conduct extensive experiments to illustrate the accuracy and efficiency of our techniques. A large portion of data available on the web is present in the so called "deep web". The deep web (or invisible web or hidden web) is the name given to pages on the World Wide Web that are not part of the surface web (pages indexed by common search engines). It consists of pages which are not linked to other pages (e.g., dynamic pages which are returned in response to a submitted query). The deep web is believed to contain 500 times more data than the surface web [5] . A major part of data present on the hidden web lies behind form like interfaces. These form based interfaces are based on a back end proprietary database and a limited top-k query interface. The query interface is generally represented as a web form that takes input from users and translates them into SQL queries. These queries are then presented to the proprietary database and the top-k results provided to the user on the browser. Many online resource locater services are based on this model. To illustrate this scenario let us _______________________________ * Partly supported by a grant from Microsoft Research Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. consider the example of a generic restaurant finder service:. Example 1: Consider a web based form which lets the user choose from a set of attributes {A 1 ,A 2 ,A 3 ,...A m } where each attribute may be Boolean, categorical or numeric, e.g., cuisine, price, distance, etc. The user chooses desired values for one or more of the attributes presented (or ranges in the case of numeric attributes). This results in a query that is executed against a backend restaurant database where each tuple represents a restaurant. The top-k answers (according to a ranking function) returned from the database are then presented to the user.. The principle problem that we consider in this paper is: given such a restricted query interface, how can one efficiently obtain a uniform random sample of the backend database by only accessing the database via the public front end interface?. Database Sampling is the process of randomly selecting tuples from a dataset. Database sampling has been used in the past to gather statistical information from databases. It has a wide range of applications for the owner of the database. Given complete and unrestricted access to the database, many methods have been developed for efficiently selecting a random sample of the tuples of the database [14, 17] .. However, in today's world where most databases are present behind a proprietary curtain, there needs to be some way to obtain statistical information about the underlying data with all these restrictions in place. This information can be then used to obtain insight into the data. Statistics about a third party database can be used to obtain quality, freshness and size information inside web sources. It can be used to identify uniformity or biases of topics. A typical example is as follows: Consider a web meta-service which retrieves data on restaurants in a city. It fetches information from two or more web sources which provide data on restaurants matching certain requirements. This service gets a query from the user and then feeds it to the two or more participating restaurant search engines. The final result is a combined mix of restaurant data from these underlying sources. Knowledge of the underlying databases would allow the makers of our restaurant service to call the database with the best quality and data distribution (in relation to a specific query) preferentially over the other sources.. However, generating random samples from hidden databases presents significant challenges. The only view available into these databases is via the proprietary interface that allows only limited access -e.g., the owner of the database may place limits on the type of queries that can be posed, or may limit the number of tuples that can be returned, or even charge access costs, and so on. The traditional random sampling techniques that have been developed cannot be easily applied as we do not have full access to the underlying tables.. In this paper we initiate an investigation of this important problem. For simplicity, we consider mainly single-table databases with Boolean, categorical or numeric attributes, where the front end interface allows queries in which the user can specify values of ranges on a subset of the attributes, and the system returns a subset (top-k) of the matching tuples, either according to a ranking function or arbitrarily, where k is a small constant such as 10 or 100.. Our main result is an algorithm called HIDDEN-DB-SAMPLER, which is based on performing random walks over the space of queries, such that each execution of the algorithm returns a random tuple of the database. This algorithm needs to be run an appropriate number of times to collect a random sample of any desired size. Note that this process may repeat samples -i.e., we produce samples "with replacement". There are two main objectives that our algorithm seeks to achieve:. • Quality of the sample: Due to the restricted nature of the interface, it is challenging to produce samples that are truly uniform. Consequently, the task is to produce samples that have small skew, i.e., samples that deviate as little as possible from the uniform distribution.. • Efficiency of the sampling process: We measure efficiency of the sampling process by the number of queries that need to be executed via the interface in order to collect a sample of a desired size. The task is to design an efficient procedure that collects a sample of the desired size as efficiently as possible.. Our algorithm is designed to achieve both goals -it is very efficient, and produces samples with small skew. The algorithm is based on three main ideas: (a) Early termination: Often, a random walk may not lead to a tuple. To prevent wasted queries, our algorithm is designed to detect such events as early as possible and restart a fresh random walk; (b) Ordering of attributes: The ordering of attributes that guides the random walk crucially impacts quality as well as efficiency -we show that for Boolean databases a random ordering of attributes is preferable over any fixed order, whereas for categorical databases with large variance among the domain sizes of the attributes, a fixed ordering of attributes (from small domains to large domains) is preferable for reducing skew; (c) Parameter to tradeoff skew versus efficiency: Since sample quality and sampling efficiency are contradictory goals, our algorithm is equipped with a parameter that can be tuned to provide tradeoffs between skew and efficiency.. A major contribution of this paper is also a theoretical analysis of the quantitative impact of the above ideas on improving efficiency and reducing skew. We also describe a comprehensive set of experiments that demonstrate the effectiveness of our sampling approach.. The rest of this paper is organized as follows. In Section 2 we formally specify the problem and describe a simple but inefficient random-walk based strategy that forms the foundation of our eventual algorithm. Section 3 is devoted to the development of HIDDEN-DB-SAMPLER for the special case of Boolean databases. In Section 4 we extend the algorithm for other types of data as well as other query interfaces. Section 5 discusses related work, and Section 6 contains a detailed experimental evaluation of our proposed approach. We conclude in Section 7. In this paper we have initiated an investigation of the problem of random sampling of hidden databases on the web. We propose random walk schemes over the query space provided by the interface to sample such databases. We gave simple methods for the sampling and provided some theoretical analysis of the quantitative impact of the ideas on improving efficiency and quality of the resultant samples. We also described a comprehensive set of experiments that demonstrate the effectiveness of our sampling approach. Traditionally database sampling has been used to reduce the cost of retrieving data from a DBMS. Random sampling mechanisms have been studied in great detail e.g., [4, 8, 14, 15 and 17] . Applications of random sampling include estimation methodologies for histograms and approximate query processing using techniques (see tutorial in [6] ).. However, these techniques do not apply to a scenario where there is an absence of direct access to the underlying database. A closely related area of sampling from a search engines index using a public interface has been addressed in [2] and more recently [1] . The technique proposed by [1] , introduces the concept of a random walk on the documents on the World Wide Web using the top-k results from a search engine. However, this document based model is not directly applicable to hidden databases. In contrast to the database scenario, the document space is not available as a direct input in the web model. This leads to the use of estimation techniques which work on assumptions of uniformity of common words across documents. Random sampling techniques on graphs have been implemented using Markov Chain Monte Carlo techniques, e.g., Metropolis Hastings [12, 7] techniques and Acceptance/Rejection technique [13] .. Hidden databases represent a major part of the World Wide Web and are commonly referred to as the hidden web. The size and nature of the hidden web has been addressed in [5] , [9] , [10] and [11] . Probing and classification techniques on textual hidden models have been addressed by [3] while techniques on crawling the hidden web were studied by [16] .

O B I O O B B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O B I O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Query forwarding in geographically distributed search engines Query forwarding is an important technique for preserving the result quality in distributed search engines where the index is geographically partitioned over multiple search sites. The key component in query forwarding is the thresholding algorithm by which the forwarding decisions are given. In this paper, we propose a linear-programming-based thresholding algorithm that significantly outperforms the current state-of-the-art in terms of achieved search efficiency values. Moreover, we evaluate a greedy heuristic for partial index replication and investigate the impact of result cache freshness on query forwarding performance. Finally, we present some optimizations that improve the performance further, under certain conditions. We evaluate the proposed techniques by simulations over a real-life setting, using a large query log and a document collection obtained from Yahoo!. Categories and Subject DescriptorsAlgorithms, Design, Performance, ExperimentationSearch engines, distributed IR, query forwarding, optimization, linear programming, index replication, result cachingCommercial web search engines of the past relied on a single search site (data center), which processed queries issued Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. from all around the world. This approach had the typical scalability problems in centralized architectures. Moreover, queries issued from distant locations suffered from poor response times as the network latency between the user and the site became an issue. For such queries, either the query processing times had to be shortened, thus degrading the result quality, or users experienced unreasonable response times, which had implications on user satisfaction [16] .At this point, replicating the data (i.e., the web collection and the inverted index built upon it) over multiple, geographically distant search sites emerged as a feasible solution. In this strategy, each geographical region is mapped to a nearby search site. A search site processes over its full web index only the queries originating from the regions assigned to itself 1 . Although this strategy reduces network latencies, the scalability still remains as an issue since the entire web index had to be maintained on all search sites and queries are evaluated over the full web index.A strategy that contrasts replication is to partition the data disjointly and assign each site only the documents obtained (crawled) from its region [8] . In this strategy, local queries of a region are evaluated over the partial index in the corresponding search site. The underlying assumption here is that users are interested more in documents located in their own region and local documents are more relevant for queries originating from the same region. As queries are now evaluated over small subsets of the web index, gains are possible in query processing time and throughput [8] , along with other gains, such as those in web crawling [9] .Unfortunately, although the assumption about having high relevance between the documents and queries of the same region is reasonable, this is not true for all queries as some queries target non-regional documents [4] . This implies that evaluation over a partitioned index will lead to inferior search quality as some relevant, non-local documents are not retrieved. The problem of accessing non-local documents has two immediate solutions: taking the data to where it is sought and/or taking the queries to what they seek. The first is an offline solution that requires partial replication of the popular documents in a region on some non-local search sites. The second is an online solution that requires selective 1 Herein, we refer to such queries as local queries.90 We showed that the fraction of locally processed queries in a multi-site search engine can be significantly increased by using an LP-based thresholding technique, and results are further improved by caching and replication. There are three research directions surfaced by our work. First, the applicability of our techniques to other problems (e.g., tiering [5] ) should be investigated. Second, a trade-off analysis is needed between forwarding performance and offline query generation and storage overheads. Finally, the freshness of precomputed score thresholds needs further research. Although there is much research on distributed IR [1] , little research is done on multi-site, distributed search engines [3, 8] . Cambazoglu et al. present cost models and results, showing the potential of multi-site architectures for efficiency and relevance improvements [8] . Baeza-Yates et al. develop analytical models to compare operational costs of multi-site search systems against centralized systems [3] .. The work in [3] is the closest to ours in that it also proposes an algorithm that tries to increase the number of locally processed queries by a thresholding technique, based on precomputation of maximum score contributions for all terms in the global vocabulary and replicating this information on all search sites. This way, it becomes possible to locally compute the maximum score a document can get on a non-local site, simply summing the maximum possible scores for query terms without evaluating the query remotely at all. It turns out that this technique is a limited case of our general solution (the same as using D1 in our setting).. We note that the query forwarding problem we deal with is somewhat different than the collection selection problem in federated IR [6] . In our system, queries are evaluated over a local index and some of them are forwarded. In federated IR, all queries are forwarded without any evaluation on a local index. We further note that P2P search systems [17] are also very different due to existence of a very high number of peers, their volatile nature, and limited availability. BaezaYates et al. [5] describe an architecture in which the global index is split into two tiers. In this architecture, queries are evaluated on one or two tiers, based on the decision of a machine-learned corpus predictor. In that architecture, documents are split into tiers by their importance or location.. Finally, Das et al. employ an LP-based solution in the context of databases for top k computation using materialized views [10] . Kumar et al. apply a similar technique to generalize top k thresholding algorithms by using precomputed intersections of posting lists [13] . To our knowledge, there is no work applying a similar technique in our context.

B I O O O B I B I O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B I B B I B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Capacity and Error Probability Analysis for Orthogonal Space-Time Block Codes over Correlated Rayleigh and Rician Fading Channels In this paper, the capacity and error probability of orthogonal space-time block codes (STBCs) are presented for PAM/PSK/QAM modulation in correlated flat fading channels. We consider an equivalent scalar AWGN (additive white Gaussian noise) channel with a channel gain proportional to the Frobenius norm of the matrix channel. A unified approach to the error probability analysis for correlated Rayleigh and Rician fading channels is presented. Closed form error probability expressions are derived for Rayleigh fading channels. We also determine the capacity and probability of error for a multiuser direct sequence code division multiple access (DS-CDMA) system employing a STBC over correlated fading channels. In recent years, orthogonal space-time block codes (STBCs) [1] - [3] have been widely employed to provide diversity gain with very low computational complexity. The major drawback is a loss in capacity, as shown in [4] for an arbitrary channel. An equivalent scalar Additive White Gaussian Noise (AWGN) channel was given in [4] assuming a full code rate. Similar work can also be found in [5] , in which the closed form capacity of several diversity systems in Rayleigh fading channels was obtained. However, the results in [5] were derived for adaptive transmission with diversity combining, while the results in this paper can be generalized to arbitrary modulation schemes and communication systems.. The Shannon capacity for an AWGN channel with continuous-valued inputs and outputs is C = W log 2 (1 + S NR), where S NR is the signal-to-noise ratio and W is the channel bandwidth. However, a channel employing a STBC with PAM/PSK/QAM modulation has discrete-valued inputs and continuous-valued outputs, which imposes additional constraints on the capacity. In [6] , a new capacity expression for independent fading channels was introduced which takes into account the constraint of discrete-valued inputs, as well as the capacity loss incurred by employing a STBC. In this paper, we extend the results in [6] capacity for STBCs in correlated fading channels. In particular, a closed form Shannon capacity expression is derived for correlated Rayleigh fading channels. The capacity loss due to the channel correlation is determined, which can aid in the design of STBC systems. Note that only orthogonal STBCs are considered in this paper. In [7] , a unified approach was presented to calculate the error rates of linearly-modulated signals over generalized fading channels based on the moment generating function (MGF) of the SNR for multichannel reception. A similar analysis was given in [8] for binary maximum ratio combining (MRC) multichannel reception. In [9] , equal gain diversity combining over a Nakagami fading channel was discussed. In [10] , a closed form error probability expression for MRC over independent Rician fading channels was derived as a series expansion that can be evaluated with controlled accuracy. In [6] , the error probability was analyzed from the SNR perspective based upon the equivalent scalar channel induced by the STBC for independent fading channels. In this paper, we continue the work of [6] by extending the analysis for PAM/PSK/QAM modulation to correlated fading channels. A unified approach is introduced for various fading channels and modulation schemes, and this is used to obtain error probability expressions for Rayleigh and Rician fading channels.. The rest of the paper is organized as follows. Section 2 presents STBCs and the equivalent channel model. The capacity analysis over correlated fading channels is given in Sect. 3. In Sect. 4, error probabilities are derived for different modulation schemes and correlated fading channels. In Sect. 5, the error probabilities and capacity analysis are extended to DS-CDMA systems employing a STBC. Section 6 presents some numerical results to illustrate and verify the analysis given in the previous sections, and Sect. 7 provides some conclusions. A comprehensive analysis of the capacity and error probability of orthogonal space-time block codes in correlated fading channels has been presented. Correlated Rayleigh and Rician fading channels were considered for PAM, PSK and QAM modulation. Closed form Shannon capacity expressions were derived for a STBC in correlated Rayleigh fading channels. Channel capacities for STBCs with PAM/PSK/QAM modulation over correlated fading channels were also investigated. Closed-form error probabilities were derived for correlated Rayleigh fading channels and various modulation techniques. A unified approach to the error probability analysis was also introduced for correlated Rayleigh and Rician fading channels. A simple expression was obtained based on the characteristic function of the instantaneous SNR. Using this analysis, the performance of a STBC multiuser DS-CDMA system was determined.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O


Compiler-generated communication for pipelined FPGA applications In this paper, we describe a set of compiler analyses and an implementation that automatically map a sequential and un-annotated C program into a pipelined implementation, targeted for an FPGA with multiple external memories. For this purpose, we extend array data-flow analysis techniques from parallelizing compilers to identify pipeline stages, required inter-pipeline stage communication, and opportunities to find a minimal program execution time by trading communication overhead with the amount of computation overlap in different stages. Using the results of this analysis, we automatically generate application-specific pipelined FPGA hardware designs. We use a sample image processing kernel to illustrate these concepts. Our algorithm finds a solution in which transmitting a row of an array between pipeline stages per communication instance leads to a speedup of 1.76 over an implementation that communicates the entire array at once. The use of pipelined execution is an effective method to improve the throughput of a program. FPGA-based computing machines offer a unique opportunity for the realization of custom pipelining structures, matching the definition of the pipeline to the application requirements in terms of pipeline stage definitions, computation that can be overlapped, the data and rates at which it can be communicated as well as the communication placement within the pipeline stages.. The complexity and sophistication of pipelined execution make automatic tools that can analyze sequential applications and derive pipelined implementations extremely desirable. In this paper, we describe a set of compiler analyses to derive, from a sequential algorithm description, the components of a pipelined design, Le., task parallelism and communication requirements. We use as a foundation parallelizing compiler analyses for array data-flow analysis [3, 61, which we extend to recognize pipelining opportunities and derive communication requirements, for use in mapping to FPGA systems. In the Design Environment for Adaptive Computing Technology (DEFACTO) [8] , we combine these analyses with behavioral synthesis tools to automatically synthesize application-specific pipelines onto a target FPGA-based architecture. The work described in this paper makes the following specific contributions:. It defines new analyses for characterizing the task parallelism and communication requirements for use in mapping sequential programs t o systems of configurable logic.. It describes an implementation of the analyses and code transformations required t o automatically design and synthesize pipelines tailored to sequential program characteristics.. It presents experimental results for a machine vision application excerpt (MVIS) which demonstrate the use and performance potential of these techniques on an FPGA-based architecture.. The paper is organized as follows. In section 2 we describe the problem we are addressing along with an example that illustrates the approach. Section 3 describes the compiler analysis in more detail. We present the execution times for four communication schemes in section 4. In section 5 we survey related work and conclude in section 6. In this paper, we describe how parallelizing compiler technology can be adapted and integrated with hardware synthesis tools, to automatically derive, from sequential C programs, pipelined implementations for systems with multiple FPGAs and memories. We describe our implementation of these analyses in the DEFACTO system, and demonstrate this approach with a case study, a machine vision application. We presented experimental results, derived automatically by our system. We illustrate how these analyses can improve application performance, as evidenced by the 1.76 speedup we gain over a non-pipelined implementation. Current work focuses on integrating these analyses with automated design space exploration for a single loop [13] and partitioning pipelined implementations over multiple FPGAs. For MVIS, we choose minDim = 1, since a row-sized transmission, described by Previous work on array data flow analysis [6, 14, 31 focused on data dependence analysis but not at the level of In [5] Arnold created a software environment to program a set of FPGAs connected to a workstation. [9] focused on compiling for a tightly coupled hybrid FPGA and RISC architecture; Callahan and Wawrzynek [7] used a VLIW-like compilation scheme for the GARP project; both works exploit intra-loop pipelined execution techniques. Goldstein et al. [lo] describes a custom device that implements an execution-time reconfigurable fabric. Weihardt and Luk [15] describes a set of program transformations to map the pipelined execution of loops with loop-carried dependences onto custom machines.. The approach taken in this paper differs from previously mentioned efforts. Our approach takes un-annotated sequential programs and maps them into a pipelined execution scheme without programmer intervention. Unlike concurrent languages, our approach neither relies on nor exploits concurrent specification behavior. Instead of focusing on intra-loop pipelining techniques that optimize resource utilization, we focus on increased throughput through task parallelism coupled with pipelining, which we believe is a natural match for image processing data intensive and streaming applications.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O


Human-competitive results produced by genetic programming Abstract Genetic programming has now been used to produce at least 76 instances of results that are competitive with human-produced results. These human-competitive results come from a wide variety of fields, including quantum computing circuits, analog electrical circuits, antennas, mechanical systems, controllers, game playing, finite algebras, photonic systems, image recognition, optical lens systems, mathematical algorithms, cellular automata rules, bioinformatics, sorting networks, robotics, assembly code generation, software repair, scheduling, communication protocols, symbolic regression, reverse engineering, and empirical model discovery. This paper observes that, despite considerable variation in the techniques employed by the various researchers and research groups that produced these human-competitive results, many of the results share several common features. Many of the results were achieved by using a developmental process and by using native representations regularly used by engineers in the fields involved. The best individual in the initial generation of the run of genetic programming often contains only a small number of operative parts. Most of the results that duplicated the functionality of previously issued patents were novel solutions, not infringing solutions. In addition, the production of human-competitive results, as well as the increased intricacy of the results, are broadly correlated to increased availability of computing power tracked by Moore's law. The paper ends by predicting that the increased availability of computing power (through both parallel computing and Moore's law) should result in the production, in the future, of an increasing flow of human-competitive results, as well as more intricate and impressive results. The goal of getting computers to automatically solve problems is central to artificial intelligence, machine learning, and the broad area of research encompassed by what Alan Turing called ''machine intelligence'' [1, 2] .. As early as 1948, Turing recognized the possibility of employing the processes of natural selection and evolution to achieve machine intelligence. In his essay ''Intelligent Machines,'' Turing [1] identified three approaches for creating intelligent computer programs.. The first approach was a logic-driven search. Turing's interest in this approach is not surprising in light of Turing's own pioneering work in the 1930s on the logical foundations of computing.. The second approach for achieving machine intelligence was what Turing called a ''cultural search'' in which previously acquired knowledge is accumulated, stored in libraries, and brought to bear in solving a problem-the approach taken by subsequent work in the field of knowledge-based expert systems.. The third approach that Turing identified in 1948 for achieving machine intelligence is:. ''… the genetical or evolutionary search by which a combination of genes is looked for, the criterion being the survival value.'' In his 1950 paper ''Computing Machinery and Intelligence,'' Turing described how evolution and natural selection might be used to automatically create an intelligent computer program [2] .. ''We cannot expect to find a good child-machine at the first attempt. One must experiment with teaching one such machine and see how well it learns. One can then try another and see if it is better or worse. There is an obvious connection between this process and evolution, by the identifications ''Structure of the child machine = Hereditary material'' ''Changes of the child machine = Mutations'' ''Natural selection = Judgment of the experimenter'' Thus, Turing correctly perceived in 1948 and 1950 that machine intelligence might be achieved by an evolutionary process in which a description of a computer program (the hereditary material) undergoes progressive modification (mutation) under the guidance of natural selection (i.e., selective pressure in the form of what is today usually called ''fitness'' by practitioners of genetic and evolutionary computation). However, Turing did not envision a population of entities being involved in the search process or any analog of sexual recombination.. In the 1975 book Adaptation in Natural and Artificial Systems, John Holland formalized the notion of ''combination [s] of genes'' and employed a population of entities in a genetic search [3] .. Genetic programming is an extension of John Holland's genetic algorithm to the domain of computer programs. Specifically, genetic programming breeds an everimproving population of programs by iteratively transforming a population of computer programs into a new generation of programs using the Darwinian principle of natural selection and analogs of naturally occurring genetic operations such as recombination (crossover), mutation, gene duplication, gene deletion, and mechanisms of developmental biology [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] .. The main points of the 1992 book Genetic Programming: On the Programming of Computers by Means of Natural Selection [6] were that. • Virtually all problems in artificial intelligence, machine learning, adaptive systems, and automated learning can be recast as a search for a computer program.. • Genetic programming provides a way to successfully conduct the search for the desired computer program in the space of computer programs.. Genetic programming developed from the seminal work of numerous researchers in the 1970s and 1980s. Holland [3] discussed the possibility of using the genetic algorithm to evolve sequences of assembly code. In 1978, Holland also proposed a broadcast language in which the genetic algorithm operated on structures more complex than fixed-length character strings. Holland and Reitman [16, 17] introduced the genetic classifier system in which sets of if-then logical production rules were evolved by means of a genetic algorithm. In 1980, Stephen F. Smith [18] introduced the variable-length genetic algorithm and applied it to populations consisting of a hierarchy of if-then rules. In particular, Smith introduced a crossover operation for populations of if-then rules. In an often-overlooked 1981 paper, Forsyth [19] introduced an innovative system called BEAGLE (Biological Evolutionary Algorithm Generating Logical Expressions) in which logical expressions were evolved in an evolutionary process. In the mid-1980s, Nichael Cramer [20] described innovative experiments in program induction employing Smith's crossover operation [18] . Hicklin (a student of John Dickinson at the University of Idaho) described a system with mutation and reproduction of computer programs [21] ; Fujiki (another of Dickinson's students) applied all genetic operations (including a crossover operation) to logical programs [22] ; Fujiki and Dickinson performed induction of if-then clauses for playing the iterated prisoner's dilemma game [23] ; Antonisse and Keller applied genetic methods to higher-level representations [24] ; and Bickel and Bickel applied genetic methods to if-then expert system rules [25] .. Since this early research in genetic programming, the field has grown significantly, and work in the field of genetic programming is regularly reported in numerous publications and conferences, including the journal Genetic Programming and Evolvable Machines; the journal IEEE Transactions on Evolutionary Computation; the annual Genetic and Evolutionary Computation Conference (GECCO) [26] combining the former Genetic Programming Conference held between 1996 and 1998 [27] [28] [29] and the former International Conference on Genetic Algorithms held between 1985 and 1997 [30] ; the annual Euro-Genetic Programming conference [31] ; the annual Genetic Programming Theory and Applications workshop [32] ; the Asia-Pacific Workshops on Genetic Programming [33] ; edited collections of papers such as the three Advances in Genetic Programming books [34] [35] [36] ; and in various other journals and conferences in the field of genetic and evolutionary computation. In addition, applications of genetic programming appear in publications devoted to the subject matter of the work. Additional sources of information about genetic programming, including links to available software for implementing genetic programming, can be found at www.genetic-programming.org and at the Genetic Programming bibliography at http://www.cs.bham.ac.uk/*wbl/biblio. The aim of the field of machine intelligence is, to paraphrase Arthur Samuel [37] from the 1950's:. ''How can computers be made to do what needs to be done, without being told exactly how to do it?''. The criterion for success for the field of machine intelligence is often couched in terms of comparisons with results produced by humans. For example, Samuel [38] stated in 1983, ''The aim [is] to get machines to exhibit behavior, which if done by humans, would be assumed to involve the use of intelligence.''. We say that a result produced by an automated method of machine intelligence is ''human-competitive'' if it satisfies one of the following eight criteria [13] :. (A) The result was patented as an invention in the past, is an improvement over a patented invention, or would qualify today as a patentable new invention. (B) The result is equal to or better than a result that was accepted as a new scientific result at the time when it was published in a peer-reviewed scientific journal. (C) The result is equal to or better than a result that was placed into a database or archive of results maintained by an internationally recognized panel of scientific experts. (D) The result is publishable in its own right as a new scientific resultindependent of the fact that the result was mechanically created. (E) The result is equal to or better than the most recent human-created solution to a long-standing problem for which there has been a succession of increasingly better human-created solutions. (F) The result is equal to or better than a result that was considered an achievement in its field at the time it was first discovered. (G) The result solves a problem of indisputable difficulty in its field. (H) The result holds its own or wins a regulated competition involving human contestants (in the form of either live human players or human-written computer programs). Table 1 shows 76 instances of work (of which the author is currently aware) where genetic programming has produced a result that can be called ''human competitive'' in accordance with the above criteria. As can be seen, these 76 results come from a wide variety of fields, including quantum computing circuits, analog electrical circuits, antennas, mechanical systems, controllers, game playing, finite algebras, photonic systems, image recognition, optical lens systems, mathematical algorithms, cellular automata rules, bioinformatics, sorting networks, robotics, assembly code generation, software repair, scheduling, communication protocols, symbolic regression, reverse engineering, and empirical model discovery. In addition, there are 31 instances (shown in column 6 of Table 1 ) where the human-competitive result produced by genetic programming duplicated the functionality of a previously patented invention, infringed a previously issued patent, or created a patentable new invention. These include one instance where genetic programming has created an entity that either infringes or duplicates the functionality of a previously patented 19th-century invention, 21 instances where genetic programming has done the same with respect to previously patented 20th-century inventions, 7 instances where genetic programming has done the same with respect to previously patented 21st-century inventions, and two instances where genetic programming has created a patentable new invention [13, 66] . The first invention of these patentable new inventions covers improved tuning rules for PID (proportional, integrative, and derivative) controllers. The second is for generalpurpose (non-PID) controllers that outperform controllers based on the widely used Ziegler-Nichols tuning rules [117] and the Å ström-Hägglund tuning rules [118] .. Despite considerable variation in representation and techniques employed by the various research groups that produced the work shown in Table 1 , many of these human-competitive results shared several common features.. • Developmental genetic programming was used to produce many of the results (Sect. 2).. • Native representations were used to produce many of the results (Sect. 3).. • The best individual in early generations often contains only a very small number of operative parts (Sect. 4).. • Most of the results that duplicated the functionality of previously issued patents were novel solutions (Sect. 5).. • The production of human-competitive results, as well as the increased intricacy of the results, are broadly correlated to increased availability of computing power suggested by Moore's law (Sect. 6).. Section 7 discusses future possibilities for producing additional human-competitive results using genetic programming in line with the increased availability of computing power tracked by Moore's law and made possible through parallel computing.

B I O O B I O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O B I


Interactive painterly stylization of images, videos and 3D animations We introduce a real-time system that converts images, video, or 3D animation sequences to artistic renderings in various painterly styles. The algorithm, which is entirely executed on the GPU, can efficiently process 512 2 resolution frames containing 60,000 individual strokes at over 30 fps. In order to exploit the parallel nature of GPUs, our algorithm determines the placement of strokes entirely from local pixel neighborhood information. The strokes are rendered as point sprites with textures. Temporal coherence is achieved by treating the brush strokes as particles and moving them based on optical flow. Our system renders high quality results while allowing the user interactive control over many stylistic parameters such as stroke size, texture and density. Artists over hundreds of years have refined a range of painting techniques to convey a scene while injecting their own individual style as a vehicle for abstraction, expressiveness and creativity. The most visually distinctive feature in many painting styles is the clear outline of individual brush marks on the canvas. Researchers working on non-photorealistic rendering (NPR) have introduced a variety of computer graphics techniques for painterly rendering wherein brush strokes emulate many of the effects seen in traditional paintings. The bulk of such research has sought to optimize the brush paths or overall arrangement of the strokes, often at the expense of substantial computation. This paper presents a system for stylization of images, video, and 3D models. The method supports a broad range of painterly styles based on brush stroke primitives, via a toolbox of parameters that control, for example, stroke size or density. To facilitate the cycle of experimentation and observation, it is crucial to offer the user interactive control over such parameters. Moreover, a fully interactive system supports applications where the input data is not known in advance, for example games or streaming video.. In order to achieve interactive frame rates, the algorithms we describe are implemented entirely on the GPU. The challenge is to find algorithms that can exploit the parallel computing power available in this environment. Our key observation is that it is possible to create high-quality painterly renderings by making purely local decisions about the arrangement of strokes. Rather than asking the question "Where should I put the next stroke?" our approach is to ask "Should I place a stroke here?" Our strategy for answering this question is based on two components: a rendered buffer that tracks stroke density throughout the image, and stochastic processes for placing new strokes where the density is too low or deleting strokes where density is too high. These purely local processes are suitable for parallelization at the stroke level and can thus be mapped onto the GPU.. This framework easily accommodates moving imagery -either video or animated 3D models. In such cases the challenge is to maintain temporal coherence for the strokes, avoiding flickering without falling prey to the "shower door effect" (the illusion that the image is seen through a semi-transparent shower door whose facets are the set of strokes fixed in the image plane). Our approach is to transport the strokes according to optical flow (for video) or exact geometric flow (for 3D models). As a result of stroke advection, their local density changes from frame to frame. Thus strokes are added or deleted to maintain a target density. Except for the choice of optical flow methods, the same simple pipeline works for all three media, even in combination (for example 3D models composited over streaming video). This paper and the accompanying video demonstrate the algorithm, revealing imagery in a variety of styles and allowing the user to modify parameters and display moving imagery at interactive frame rates (Figure 1 ). Applications for this work include artistic control in image and video processing applications; painterly rendering for games, virtual worlds or architectural/design tools; and stylistic range in a whimsical variation on video conferencing. In this section we provide a detailed overview of our algorithm. We introduce a set of data structures and algorithms that allow all the steps of the stylization process to take advantage of the parallelism provided by the GPU. The entire stroke generation, manipulation and deletion process uses local operations within GPU programmable shaders. In particular, we make heavy use of geometry shaders and their streaming functionality to generate and update the strokes. Our GPU-based stroke rendering system provides high quality stylized image, video and 3D animation results in various artistic styles at interactive rates. We design a parallel multi-layer strokebased stochastic approach to guide the generation and deletion of brush strokes. The movement of brush strokes and the change of stroke properties are modeled based on optical flow and geometry reprojection entirely on the GPU. We apply stroke texture and alpha masks to determine the stroke surface detail and stroke shape which, in turn, determine the feeling and style of the final rendering.. Limitations. The temporal quality of our video processing depends on the effectiveness of the optical flow implementation; we find that failures in optical flow computation lead to distracting artifacts. For example if the optical flow result is very noisy it leads to flickering, or if the result is falsely reported as zero it induces the "shower door effect." Moreover, even when flow can be computed exactly, as with geometric input, there remains a tradeoff between faithful reproduction of input image features, and maintaining temporal smoothness. Nevertheless, our system offers the user some control with regard to this tradeoff. Finally, the stochastic process we describe offers no guarantees about stroke density, and thus strokes can cluster unnecessarily or leave holes in the canvas.. Future work. We would like to further investigate the optical flow step in order to further improve the quality of the video sequences, especially under fast motion. This kind of technology could be applied to video conferencing, where advantages might include high compression (just send the stroke information), a layer of privacy (fewer concerns about makeup, hair, clothing, etc), and a fun channel for expressiveness. Moreover, a variation in this space would include virtual meetings in which parties interact with 3D objects in 3D scenes, all of which can be rendered with consistent style. For such applications it would be ideal to use face tracking methods and specialize the painterly algorithms for rendering faces. Finally, we would like to offer explicit user control over the "gradient" field or the "optical flow" field as ways of orchestrating the stroke directions or movement. This would allow for composing a picture like van Gogh's Starry Night (1889) or even a version in which the strokes swirl over time.. (a) Figure 10: Example stylized images produced using our method, together with many of the parameters used to create them. (See Section 7.1 for an overview.) We offer several examples of stylized images generated with our system. Figure 1b-d shows results based on video and animation. The accompanying video demonstrates these examples in motion, as well as showing some of the effects of interactive user control. Figure 7 shows the impact of stroke size on the rendering style of a given image, while Figure 9 shows the effect of texture choice. Figure 10 provides examples of how other stroke parameters affect the rendering style. Next we review the user controls that are available and were employed to produce such results. We then discuss the overall performance. Stroke-based rendering techniques. Brush strokes are commonly used for simulating various artistic styles. Stroke based approaches, such as [Strassmann 1986; Hertzmann 1998; Kalnins et al. 2002; Park and Yoon 2008] model the brush strokes as spline curves. One advantage is that they can model long, continuous brush strokes with varying size and shape. Model-based approaches such as [Meier 1996; Kaplan et al. 2000; Haller and Sperl 2004; Luft and Deussen 2006] use particles in 3D to model the brush strokes. Usually, they associate 3D particles with the geometry of the model, derive the stroke properties from surface attributes, and render the particles as brush strokes in screen space. These approaches are not appropriate for the image and video domain. First, modeling spline curves on the GPU introduces added complexity with a substantial performance impact. Second, model-based approaches make use of 3D geometry properties that are not available in 2D image domain. In our work, we represent brush strokes as particles in the image and time domain and determine the particle properties based solely on image processing. Thus, our approach has the flexibility to handle image, video, and 3D geometry.. Previous researchers have introduced several stroke-based rendering algorithms for image processing. For example, Shiraishi and Yamaguchi [2000] aimed to create an automatic painterly rendering system with minimal user intervention. They estimate the stroke properties based by approximating the local regions of the source image with rectangular brushes. Gooch et al. [2002] presented a method to use the approximate medial axes of the segmented features of the image to guide the creation of brush strokes. Kovács and Szirányi [2004] proposed a fully automatic rendering method that targets at removing randomness and providing a more natural look by using the image features to guide all of the parameters. All these methods rely on the heavy use of local image features and require expensive computation. Our algorithm makes the decision to place strokes independently at each location, which simplifies the rendering process and still produces high quality rendering results.. Closest to our work in stroke-based rendering are the methods of Hertzmann and Perlin [2000] , and Vanderhaeghe et al. [2007] . Hertzmann and Perlin introduce a painterly video rendering system that successively paints over earlier time frames at interactive rates. Their system can also optionally use optical flow to better track scene changes, but it significantly impacts rendering time. Our approach provides a fast real-time fully parallel algorithm pipeline that runs entirely on the GPU and also efficiently handles geometry animation as input through the use of reprojection. The method of Vanderhaeghe et al. can also handle geometry input. Moreover, it achieves high quality results via a temporally coherent blue noise sampling distribution for stroke placement, but at the expense of real-time frame rates. Overall, our system provides a simpler and faster parallel solution when compared to these methods, while still being very generic in handling input composed of video, animated geometry, or both. Furthermore, the user interface provided by our system allows for very intuitive control of different rendering styles.. Temporal coherence in video rendering. Video processing is different from image processing and is usually composed of two sub-problems: (1) rendering individual frames in a specific style, and (2) maintaining temporal coherence across successive frames. Single-image algorithms often cannot be applied directly to individual video frames without inducing poor temporal coherence (usually flickering) into the resulting animation. To address this problem, one solution is to translate strokes from frame to frame using an estimated optical flow vector field [Litwinowicz 1997 ]. The source video sequence can also be processed as a spatiotemporal voxel volume [Collomosse et al. 2005] . Wang et al. [2004] developed an anisotropic kernel mean shift technique to segment the video data into contiguous volumes. Bousseau et al. [2007] presented a method that employs texture advection along lines of optical flow. Horn and Rhunck [1981] introduced "The Smoothness Constraint" to solve the optical flow vector field. For its simplicity, we adopt the GPU implementation of Warden [2005] .. Artistic styles in video rendering. Various styles have been realized in video processing results such as painterly [Litwinowicz 1997; Hays and Essa 2004; Park and Yoon 2008] , watercolor [Bousseau et al. 2007 ], cartoon [Wang et al. 2004; Winnemöller et al. 2006] and abstract [Klein et al. 2002] styles. Litwinowicz [1997] described a technique that transforms ordinary video segments into animations having an impressionist effect. Extending from this work, Hays and Essa [2004] presented a stroke-based painterly video rendering algorithm that constrains the change of stroke properties to guarantee temporal coherence. Park and Yoon [2008] also addressed the painterly rendering of video sequences with focus in using motion maps to maintain temporal coherence. These algorithms produce high quality video rendering results. However, as CPU-based offline algorithms, they are not suitable for real-time applications. Inspired by previous work, we design a GPU-based real-time algorithm that produces rendering results in various styles. We borrow Hays' idea for rendering brush strokes using stroke textures as height maps for per-pixel lighting.. Real-time video stylization. Real-time algorithms have been proposed to stylize video sequences. Klein et al. [2002] introduced an approach that treats the video as a space-time volume of image data (a "video cube"), and extended rendering techniques for video far beyond impressionism to more abstract styles (with interactive controls). Winnemöller et al. [2006] proposed an automatic abstraction framework for images or video. They first reduce contrast in low-contrast regions while enhancing contrast in higher contrast regions and then stylize the imagery using soft color quantization. Hong et al. [2008] later proposed an extension to further improve the processing efficiency. These approaches are GPU-based, but they do not model individual brush strokes, limiting the range of possible styles. Our system operates in real time without precomputation, and supports a broad range of painterly styles.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O