A General Framework for Efficient Continuous Multidimensional Top-k Query Processing in Sensor Networks Abstract-Top-k query has long been a crucial problem in multiple fields of computer science, such as data processing and information retrieval. In emerging cyber-physical systems, where there can be a large number of users searching information directly into the physical world, many new challenges arise for top-k query processing. From the client's perspective, users may request different sets of information, with different priorities and at different times. Thus, top-k search should not only be multidimensional, but also be across time domain. From the system's perspective, data collection is usually carried out by small sensing devices. Unlike the data centers used for searching in the cyber-space, these devices are often extremely resource constrained and system efficiency is of paramount importance. In this paper, we develop a framework that can effectively satisfy demands from the two aspects. The sensor network maintains an efficient dominant graph data structure for data readings. A simple top-k extraction algorithm is used for user query processing and two schemes are proposed to further reduce communication cost. Our methods can be used for top-k query with any linear convex query function. The framework is adaptive enough to incorporate some advanced features; for example, we show how approximate queries and data aging can be applied. To the best of our knowledge, this is the first work for continuous multidimensional top-k query processing in sensor networks. Simulation results show that our schemes can reduce the total communication cost by up to 90 percent, compared with a centralized scheme or a straightforward extension from previous top-k algorithm on 1D sensor data. T HE recent development of sensor networks [8] , [10] , [19] , [20] , [21] , [22] has made it possible for users to search information not only in the cyber-space, but also in the physical world. It can be imagined that in the near future people would be able to enjoy searching temperature, humidity, light, smoke of various times, in a forest, according to their own preferences. The fire service department may focus more on the temperature and smoke of the region, while the zoologists may be more interested in light and temperature. These application requirements call for a key function from the system design, an efficient processing of the continuous multidimensional top-k queries in sensor networks.. Top-k query processing has long been an important task in various research domains [29] , [39] where the k highest (or lowest) data points are retrieved from a large data set. The unique challenges we face in the aforementioned applications come from two aspects. First, from the client's perspective, there can be a large number of different users. Each of them has his own preference; they not only put different weights on different dimensions of data but also on different time periods of data. This multidimensional nature has made many algorithms [2] , [34] , [38] developed for 1D top-k queries unsuitable or inefficient. With multidimensional sensor data, they have to pose as many queries as the number of user requests since each user could assign a set of weights representing his own preference, which results in huge communication overhead. Second, from the system's perspective, the sensing devices are distributed and usually with great resource constraints. Especially, for the energy limited sensor nodes, the communication should be tightly optimized. Therefore, most centralized schemes developed in the database community such as [39] cannot be applied here. As an example, ordinary nodes have no information about the user preference and if all sensor data are extracted, it could incur a large amount of communications.. To this end, we develop a framework based on dominant graph (DG) [39] . DG is a layered data structure to build a relationship between different data points in multidimensions. To successfully apply DG in distributed sensor networks, we face many challenges. In [39] , the server extracts top-k results using the given query function. This is a pure centralized operation. In a distributed sensor network, unfortunately, it is impossible for the sensor nodes to know the query functions since the query is performed at the sink. To successfully carry out the top-k query in sensor networks, there must be interactions between the sink and the sensors in the network. The incurred communication traffic dominates the energy consumption, and is of crucial importance for system efficiency, thus it requires special treatment.. We believe the best way to handle such difficulties is to develop a framework for the sensor network. In the framework, the complexity of multidimensional top-k query processing and system efficiency can be clearly assigned to sinks and the sensor nodes, respectively. In this paper, we discuss our framework and the associated algorithms. To the best of our understanding, we are the first to develop solutions for continuous multidimensional top-k query processing in sensor networks. Our contributions are summarized as follows:. . We propose a novel framework that efficiently realizes DG in a distributed sensor network. The framework supports top-k queries for arbitrary userdefined linear convex query functions over multidimensional sensor data. . For each sensor, where the query function is unknown, we develop a top-k extraction algorithm to efficiently retrieve necessary data points that will be sent to the sink. . We propose schemes working with the top-k extraction algorithm for reducing communication traffic. Our key observation is that the continuously collected sensor data does not change abruptly. With previously generated global top-k results, we develop novel scheme for each node to update and maintain its local dominant graph for later data suppression. In addition, we propose local filters to further reduce the communication traffic. . We develop a comprehensive set of modules and interactions. Our framework is general enough so that many modules can be added piece by piece. We show two examples on how to handle data aging and accommodate approximate queries so as to adapt to different user requirements and system performance. . We evaluate our framework on both real and synthetic data sets. Simulation results show that our schemes can reduce the total communication cost by up to 90 percent, compared with the centralized scheme or a straightforward extension from previous top-k algorithm designed for 1D sensor data. The remainder of this paper proceeds as follows: Section 2 describes the background of top-k (preference) query in multidimensional data and our framework; Section 3 is devoted to the top-k query processing algorithms; in Section 4 issues related to overhead reduction and complexity analysis are discussed. We evaluate our schemes in Section 5; Section 6 presents related work and finally, Section 7 summarizes the paper and gives future plan. [30] , [34] , [38] and arrival timestamp (used in slide window queries), respectively. Let a user-defined query function be F ðd i Þ ¼ AE m j¼1 w j Á d i :xj where w j represents the weight on the jth dimension. A top-k preference query (or top-k query for short) is to retrieve k data points from D whose values of function F are the highest. Consistent with [29] , [39] , we only consider the typical linear convex query functions here. Also known as aggregate monotone functions [13] A user starts a query with: 1) a set of user-defined weights; 2) k, the number of data points to be retrieved; and 3) a time period s ! 1 during which the top-k query results should be retrieved. With a sliding window model, the topk results in the previous time window of size s is returned. This paper presents the first work on continuous multidimensional top-k query processing in wireless sensor networks. We have developed a framework which effectively monitors user queries and in-network process. More importantly, it incorporates a special data structure, the dominant graph, to maintain top-k query results. The dominant information can facilitate identifying the potential top-k query results for any given preference function and filtering out nontop-k results. The framework is general enough such that optional modules such as approximate query processing, data aging, etc., can be handled. We prove that our algorithms are scalable as the message complexity is proportional to the total number of sensor nodes. The simulation shows that our algorithms reduce the communication cost by up to 90 percent as compared to the centralized scheme and a straightforward extension from previous top-k algorithm on 1D sensor data.. We are interested in several directions in future work. First, we will seek more efficient algorithms to reduce the traffic overhead of our proposed framework. Second, evaluation of our framework on larger scale networks will be carried out. Finally, besides simulations in this paper, a testbed evaluation will be performed. . For more information on this or any other computing topic, please visit our Digital Library at www.computer.org/publications/dlib. Many studies [4] , [9] , [16] , [17] , [18] , [23] , [31] , [32] have explored various data aggregation techniques for query processing in sensor networks in order to reduce communication cost and energy consumption. Some studies strive to design energy-efficient processing methods for top-k queries. Silberstein et al. [30] proposed to use the samples of past sensor readings for query optimization and developed a series of top-k query planning algorithms with linear programming. Zeinalipour et al. [38] proposed to use nonuniform thresholds on the queried attribute in order to minimize the number of tuples transferred toward the querying node. Similarly, Wu et al. [34] proposed to use a filter at sensor nodes to suppress unnecessary sensor readings. Aiming at continuous top-k monitoring, ZeinalipourYazti et al. proposed MINT [36] , a framework that maintains the complete results of a query and calculate the upper bounds, so as to reduce the cost of future queries by pruning phase. Instead of using a shortest-path tree for routing, XP [24] employs the cluster tree as the routing infrastructure, in order to aggregate many data points locally before further transmissions. Chen et al. [7] strived to return k highest data points generated within a specified time interval.. All these studies unfortunately have focused on 1D data set. Our distributed, multidimensional, and historical query processing framework is motivated by some previous works [3] , [29] , [39] in the database community. Babcock and Olston [3] worked on the 1D and error-tolerant top-k monitoring over distributed nodes by setting up the local parameterized constrains which are calculated by the centralized node. Cao and Wang [6] proposed a uniform threshold algorithm (TPUT) to reduce the remote accesses. Besides exact algorithms, Yu et al. proposed TPAT [35] , exploiting statistics to further suppress the readings of TPUT. Michel et al. [27] considered approximate top-k queries in distributed environments where at each node the approximation consists of an histogram on the local scores along with a bloom filter. Thus, it is applicable for approximate and ondemand queries. Further, UBK/UBLB-K algorithms [37] addressed exact answers. Mouratidis et al. [29] studied centralized continuous monitoring of top-k queries over a fixed-size sliding window via partially precomputing the future changes. In [39] , a dominant graph-based algorithm was introduced for centralized, multidimensional, and snapshot query processing. In sensor networks, however, energy is of crucial importance which requires more special treatment.

O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O


Toxic Effects of Opioid and Stimulant Drugs on Undifferentiated PC12 Cells ABSTRACT: Cell death and reactive oxygen species production have been suggested to be involved in neurodegeneration induced by the drugs of abuse. In this study we analyze the toxicity of the following drugs of abuse: heroin, morphine, d-amphetamine, and cocaine in undifferentiated PC12 cells, used as dopaminergic neuronal models. Our data show that opioid drugs (heroin and morphine) are more toxic than stimulant drugs (d-amphetamine and cocaine). Toxic effects induced by heroin are associated with a decrease in intracellular dopamine, an increase in DOPAC levels, and the formation of ROS, whereas toxic effects induced by amphetamine are associated with a decrease in intracellular dopamine and in ATP/ADP levels. In contrast with cocaine, both amphetamine and heroin induced features of apoptosis. The data suggest that the death of cultured PC12 cells induced by the drugs of abuse is correlated with a decrease in intracellular dopamine levels, which can be associated with an increased dopamine turnover and oxidative cell injury. Drug addiction is associated with a repeated use of a drug or a combination of drugs. The dopaminergic system has been identified as a critical and shared pathway involved in drug reward, and the drugs prone to abuse have been shown to elevate synaptic dopamine levels in this pathway. 1 Dopamine can be oxidized by MAO B in humans, giving rise to DOPAC and H 2 O 2 . 2 Although not a free radical because it does not contain unpaired electrons, H 2 O 2 can interact with transition metal ions and produce the highly toxic hydroxyl radical ( • OH) via the Fenton-Haber Weiss reaction. Therefore, reactive oxygen species (ROS) have been frequently associated with neuronal cell death due to damage to carbohydrates, amino acids, phospholipids, and nucleic acids. Moreover, the rise in H 2 O 2 levels can be followed by a decrease in the levels of reduced glutathione (GSH), which can lead to the inhibition of mitochondrial respiratory chain activity and, subsequently, to cell death. 3 In addition, dopamine has been shown to induce cell death by apoptosis in several cell types, [4] [5] [6] and to inhibit mitochondrial respiration. 7 In the present study, using PC12 cells as models of dopaminergic neurons, 8, 9 we explored the effects of toxic concentrations of some opioid and stimulant drugs of abuse-namely, amphetamine, heroin, and cocaine-that are known to cause severe health problems in the Portuguese population. The data highly suggest that toxic effects induced by the drugs of abuse in vitro are associated with a decrease in intracellular dopamine levels, probably related with an enhanced dopamine turnover and oxidative stress. In this study we show that cell death induced by the drugs of abuse involves a decrease in intracellular dopamine content in cultured undifferentiated PC12 cells, without major changes in membrane integrity, suggesting an alteration in dopamine metabolic pathways and/or an increase in extracellular dopamine accumulation.. Cytotoxic effects induced by cocaine were not very evident in PC12 cells, as demonstrated by the maintenance of MTT reduction or ATP/ADP levels, the nonappearance of apoptotic features or the small decrease in intracellular dopamine, in contrast with heroin or amphetamine. Nevertheless, cocaine was previously shown to induce cell death by apoptosis in several cell types, including fetal mouse cortical neurons, 18 fetal rat myocardial cells, 19, 20 bovine coronary artery endothelial cells, 21, 22 or in rat testes. 23 Moreover, cocaine, and in particular its N-oxidative metabolites, were shown to depress mitochondrial respiration. 24 Cytotoxic effects induced by heroin were associated with a large decrease in intracellular dopamine, an increase in DOPAC levels, and an apparent, nonstatistical increase in intracellular ROS production. In addition, toxic effects induced by amphetamine in PC12 cells were associated with a decrease in intracellular dopamine and a decrease in ATP/ADP levels, suggesting a certain degree of metabolic dysfunction. Interestingly, both amphetamine and heroin induced chromatin condensation, a feature of apoptotic cell death, which is in agreement with previous studies showing the induction of apoptotic cell death by heroin 25 and amphetamine. 26 Furthermore, although our data show an attenuation of ATP production and the appearance of apoptotic features upon exposure to amphetamine, recent data reported by Lotharius and O'Malley 27 support the idea that amphetamine-induced toxicity in dopaminergic neurons involves production of ROS, although neither protein oxidation, ATP decrement, mitochondrial dysfunction, nor cell death was observed.. In conclusion, toxic effects of amphetamine and heroin in undifferentiated PC12 cells are associated with the induction of apoptotic cell death, which may be due to a metabolic dysfunction induced by amphetamine or an increase in dopamine metabolization and production of ROS induced by heroin. As for cocaine, further experimental approaches are needed to clarify the cytotoxic mechanisms. The results reported in this study are compatible with the hypothesis that dopamine plays a major role in reinforcing the toxic effects of the drugs of abuse.

O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O B O O B I O O O O O O O O O O O B O B O O O O O O O O B O O O O B O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O B O B O B O O O O O O O O O O O O B I O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O B I O O O O O O O O O O B O O O O O O O B O O O O O B O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B O O O B O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O B O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O B I O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O


Scalable learning of probabilistic latent models for collaborative filtering Collaborative filtering has emerged as a popular way of making user recommendations, but with the increasing sizes of the underlying databases scalability is becoming a crucial issue. In this paper we focus on a recently proposed probabilistic collaborative filtering model that explicitly represents all users and items simultaneously in the model. This model class has several desirable properties, including high recommendation accuracy and principled support for group recommendations. Unfortunately, it also suffers from poor scalability. We address this issue by proposing a scalable variational Bayes learning and inference algorithm for these types of models. Empirical results show that the proposed algorithm achieves significantly better accuracy results than other straw-men models evaluated on a collection of well-known data sets. We also demonstrate that the algorithm has a highly favorable behavior in relation to cold-start situations. Recommender systems have become a well-established technology to help users cope with vast amounts of information. This is achieved by only presenting to the users the information which is deemed most relevant. Over the last years the diversity of the domains in which recommender systems have been successfully applied has increased significantly and include movies, books, news, and products in general.. Recommender systems are typically grouped into two categories: contentbased systems make item recommendations by combining content descriptions of the items in questions with a preference model of the user (e.g. inferred using previously rated items). On the other hand, collaborative filtering systems provide recommendations based on the ratings of other users with similar preferences. The two types of systems exhibit different characteristics; collaborative filtering systems typically enjoy a greater flexibility in terms of the types of items that can be recommended whereas content-based systems are usually less susceptible to cold-start problems.. Collaborative filtering systems are often further sub-divided into model-based and memory-based [4] methods, although combinations of the two have also been proposed [29] . Memory-based systems rely on a distance measure to estimate user similarity, whereas model-based approaches learn a model of the user's preferences, which is subsequently used for making predictions. The earliest model-based approaches used a multinomial mixture model [10] for either grouping the users into user groups or items into item-categories. More recently, uniform models have been proposed that treat users and items equivalently and represent them jointly in the same model. An example of such a model is the probabilistic latent variable model proposed by Langseth and Nielsen [20] .. The model described in [20] bears some resemblances with relational probabilistic models [13] in that it explicitly combines all users and items directly in the model [34, 33, 12] . More specifically, the model is a special type of conditional linear Gaussian model [21] , where each item and user is represented by a collection of abstract latent variables encoding intrinsic properties about the user/item in question. The rating assigned to item i by user p is in turn modeled as a linear Gaussian distribution conditioned on the corresponding latent user/item representations. This joint representation of users and items allows the model to take advantage of all the user/item information available when making recommendations. Not only does this result in high-quality recommendations, as documented in [20] , but it also supports a well-founded and principled way of making group recommendations [7, 25] .. In order to learn the probabilistic latent variable models, Langseth and Nielsen [20] proposed an Expectation-Maximization (EM) algorithm tailored to the specific model class. Unfortunately, the algorithm requires the calculation of the full covariance matrix for all the latent variables representing the users and items. Consequently, the algorithm does not scale to larger data sets.. In this paper we address the scalability problem by proposing approximate learning and inference algorithms based on a variational Bayes approach [1, 3] . The algorithms employ a (generalized) mean-field approximation of the variational distribution, which ensures that the complexity of the learning algorithm grows linearly in the number of data points/ratings. Furthermore, we show that the model fits within the general class of statistical query models that in turn supports an efficient use of the MapReduce framework [8, 6] ; hence the algorithms are easily parallelizable and can exploit distributed architectures. We empirically evaluate the proposed algorithms using several well-known data sets and demonstrate that the algorithm obtains results that are significantly better than what is obtained by a collection of straw-men methods. Finally, we analyze the performance of the method under cold-start conditions [19] .. The remainder of the paper is structured as follows: In Section 2 we provide background information and describe the probabilistic latent variable model by Langseth and Nielsen [20] . Section 3 describes the variational Bayes based learning algorithm (with detailed derivations included in the appendix) and in Section 4 we present the empirical results. We conclude the paper in Section 5 and outline directions for future research. In this paper we have proposed two scalable algorithms for learning probabilistic collaborative filtering models that explicitly represents all users and items simultaneously [20] . The algorithms are based on the variational Bayes framework and differ in terms of the complexity of the variational distributions being applied. The computational complexity of the algorithms is linear in the number of ratings. Furthermore, both algorithms support a seamless parallel implementation that can easily be exploited in a MapReduce architecture. This allows for the processing of extremely large data sets, which we have illustrated by evaluating and comparing the algorithm based on the Yahoo 700M data set. The algorithms have also been evaluated on a collection of other publicly available collaborative filtering data sets and compared with well-known straw-men methods. The empirical results demonstrate that not only do the algorithms significantly outperform the straw-men methods, but we also observe a very favorable performance in cold-start scenarios. We observe only minor differences between the two algorithms wrt. prediction quality, and therefore do not find support for selecting the computationally more complex algorithm (GMF) over its simplified counterpart (MF) in our analysis. In particular, the simpler version is recommended for use in big data situations.. As part of our future work, we are currently exploring methods for extending the model and the learning algorithm to also include content information about users and items. We expect that this type of information will be encoded using discrete variables, thus producing a particular type of hybrid probabilistic collaborative filtering model. where. Further, in a Bayesian formulation of the problem, we give the following prior distributions to our parameters. This allows us to, in principle, calculate f (u, m, ρ|r). Note that we have kept µ φ = 0 fixed in the experiments reported in this paper.. Appendix A.2. The generalized mean-field model We will first assume a full variational joint distribution of the form. i.e., the distribution factors into terms so that, e.g., M i ⊥ ⊥U p |R. On the other hand, note that M i,k ⊥ ⊥M i,l |R, etc. Based on Equation (3), we get the following for M i , where i ∈ {1, . . . , #M } is fixed:. where m −i is used as a shorthand for the collection of all m j with j = i and u denotes the collection of all u p , p = 1, . . . , #U .. The integral can be expanded as:. where the constant is used to continuously collect all terms that do not depend on m i . Next, log f (r (p, i) |m i , v p , u p , w i , φ p , ψ i , θ) can be written as follows (when all terms not involving m i are continuously collected into the constant):. Consider now an r-dimensional Gaussian variable X ∼ N µ, Q −1 , where µ is the expected value and Q is the inverse variance, or precision. By simple calculation, and letting all terms that are not depending on x continuously disappear into the constant, we find that. 2), the integral in Equation (A.1) can therefore be written as. Comparing the terms of Equation (A.4) with those of Equation (A.3), we find that q (m i ) must be a Gaussian with precision. and expectation. Using the same procedure as above we find that q (v p ) also has the form of. vp , where. Similar update rules are obtained for q(u p ) and q(w i ). Next, we move on to Ψ i , which can be interpreted as the a priori rating for item i. In our Bayesian formulation,. , where µ ψ and κ ψ are the hyper-parameters, denoting the expectation and precision, respectively. Starting again from Equation (3) we have that. As before we expand the integral, and simplify by continuously moving all terms not depending on ψ i into the constant: . Thus, we get log q (ψ i ) = − 1 2 ψ. + ψ i κ ψ µ ψ + const. . Using the same procedure, we also find that q (φ p ) is a Gaussian distribution. We utilize that φ p has a priori mean µ φ = 0 to simplify the results slightly, and obtain that. Lastly, the distribution for the precision, q (θ) is to be calculated. The prior distribution of Θ is assumed to be a Gamma distribution with hyper-parameters a and b:. f (θ) = b a Γ(a) θ a−1 exp(−bθ).. As usual, we start from Equation ( and by comparing this expression with the Gamma distribution (Equation (A.5)) we find that q (θ) is a Gamma distribution with parameters: a * = a + #Obs 2. The convergence of the iterative learning scheme is controlled by monitoring the lower-bound of the marginal likelihood of the data, defined by F (q) = H(q)+ E q [log f (r, m, u, ρ)], where H(q) is the entropy of the variational distribution. The calculation of this lower-bound is straightforward given the developments above.

B I O O O O O B I B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Defining the structure of DPCA models and its impact on process monitoring and prediction activities AbstractComponents Analysis (PCA), developed in order to add the ability to capture the autocorrelative behaviour of processes, to the existent and well known PCA capability for modelling cross-correlation between variables. The simultaneous modelling of the dependencies along the "variable" and "time" modes, allows for a more compact and rigorous description of the normal behaviour of processes, laying the ground for the development of, for instance, improved Statistical Process Monitoring (SPM) methodologies, able to robustly detect finer deviations from normal operation conditions. A key point in the application of DPCA is the definition of its structure, namely the selection of the number of time-shifted replicates for each variable to include, and the number of components to retain in the final model. In order to address the first of these two fundamental design aspects of DPCA, and arguably the most complex one, we propose two new lag selection methods.The first method estimates a single lag structure for all variables, whereas the second one refines this procedure, providing the specific number of lags to be used for each individual variable. The application of these two proposed methodologies to several case studies led to a more rigorous estimation of the number of lags really involved in the dynamical mechanisms of the processes under analysis. This feature can be explored for implementing improved system identification, process monitoring and process control tasks that rely upon a DPCA modelling framework. In this paper we proposed two new methods for selecting the number of lags in DPCA models. These methods are based on the correspondence between the smallest singular values and near linear relations present on data.. The methods for selecting the maximum number of lags were compared with the procedure proposed by Ku et al. [16] and applied to a series of systems with randomly generated structures. From this analysis, we concluded that our proposed method to select the number of lags gave a closer estimation of the lags and was statistically better than the benchmark method. The same conclusion was drawn from the implementation of these methods to several models collected from the literature. We also note that, although the proposed algorithm is capable of selecting the correct stage in most of the times, it is recommend to analyse the graphical representation of the singular values, their ratio and , and select a stage with simultaneous low values for the singular value I II. A C C E P T E D M A N U S C R I P T ACCEPTED MANUSCRIPT 35 and ratio, with no subsequent lower ratio values. This situation should be considered with some attention in the first stages of the procedure, as discussed in the text.. Finally, we also concluded that the use of the proposed methods to select the number of lags, ultimately leads to superior performances in other activities based on DPCA models, such as statistical process monitoring and system identification. Future work will address the application of these methodologies to high-dimensional systems and other application scenarios. In the previous section, we have demonstrated the increased accuracy of the proposed methodologies relatively to the benchmark method proposed by Ku et al [16] , in several testing scenarios, and illustrated their added value in practice in situations where DPCA is employed for addressing SPM and SI problems. The methodologies proposed are theoretical-driven, but incorporate also results and improvements arising from an extensive analysis of possible alternatives to address the lag selection problem. This dose of empiricism is reflected in the adoption of solutions (such as the ratio restriction). that consistently led to better results, providing flexibility and robustness to the proposed methods. Other alternatives, even when grounded on a well established theoretical background, failed to provide better results, or presented implementation problems.. For instance, one approach tested to select the maximum number of lags, consisted in finding the point after which the number of new relationships are the expected ones assuming that all the relevant linear relations were extracted until the previous stage. In fact, if one is able to extract all the relationships with a certain number of lags, by adding one more lag than necessary for all variables, one should theoretically obtain all the previous extracted relationships, replicated one more time.. And this process will go on, as more lags are added. Therefore, by finding the onset of such regular behaviour, one could establish the maximum number of lags necessary to describe the dynamical behaviour of all the variables. However, such a methodology led to implementation problems that manifest in imprecise estimates of the onset of the replication process, which translate in worse results that those provided by Method I. As an example, consider the Wood and Berry system presented earlier. For this specific system, the correct number of lags is 9, and therefore, it was expected that the number of linear relations (NLR) increased in a proportional way after this lag. However, as can be seen in Figure 14 , despite the near linear relation between the NLR and the number of lags, there is no significant difference in the regions before (I) and after (II) lag 9, making it impossible to identify it as the appropriate one. In this sense, the proposed methods proved to be empirically accurate and stable in most of the circumstances studied, providing a usable solution to this non-trivial problem of model structure definition for DPCA. In this section we present the results obtained from the application of the methods presented in the previous section, to different test scenarios. In the first subsection, we demonstrate the improved accuracy in estimating the lag structure obtained with the proposed methodologies. Then, we illustrate the consequences of using such improved methods, in the tasks of statistical process monitoring (SPM) and systems identification (SI). In this section we briefly review the methods involved in the work presented in this article. As most of them are well known and extensively referred in the literature, such as PCA [31] [32] [33] and DPCA [16] , we provide just a short overview, mostly for the purpose of setting the nomenclature to be followed in the next sections.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Combined Affine Geometric Transformations and Spatially Dependent Radiometric Deformations: A Decoupled Linear Estimation Framework Abstract-This paper considers the problem of registering two observations of the same object, where the observations differ due to a combined effect of an affine geometric transformation and nonuniform illumination changes. The problem of deriving new representations of the observations that are both invariant to geometric transformations and linear in the illumination model is analyzed. In this framework, we present a novel method for linear estimation of illumination changes in an affine invariant manner, thus, decoupling the original problem into two simpler ones. The computational complexity of the method is low as it requires no more than solving a linear set of equations. The prior step of illumination estimation is shown to improve the accuracy of state-of-the-art registration techniques by a factor of two. T HE same object, viewed from different points, or under different illumination conditions, can have many different appearances. Relating the different appearances to the object and analyzing the changes in the appearance are very complex problems.. In many cases, the change of viewing point results in a geometric transformation of the observation like shift, rotation, or scale of the image coordinates. The problem of "aligning" the different coordinate systems to a single coordinate system is called image registration. Image registration is essential for integrating data from different measurements. The information derived from the geometric transformation is used in applications such as object tracking [1] and pose estimation [2] . Applications that detect changes in objects, such as in image-based quality control, require both the registration of the object and compensation for illumination changes. A survey of registration techniques is given in [3] .. In this paper, we focus on affine transformations of the coordinate system. The affine transformation consists of linear transformations (rotation, scaling, and shear) and a translation (shift). It provides a good approximation for the changes in pose of objects that are relatively far from the camera. The affine transformation also serves as the basic block in the analysis and registration of general and nonrigid geometric transformations. When the area considered is small enough, the affine transformation serves as a first-order Taylor series approximation of any differentiable geometric transformation.. The most popular methods for estimating the geometric transformations today are based on local features, such as intensity-based regions (IBR) and edge-based region (EBR) [4] , and scale-invariant feature transform (SIFT) [5] and maximally stable extremal regions (MSER) [6] . These methods identify features of small regions in the image and extract the transformation from the correspondence of the features. The correspondence problem is solved by using features that are invariant to the geometric transformation. Affine invariant features include multiscale autoconvolution (MSA) [7] , affine invariant moments [8] , cross-weighted (CW) moments [9] , and trace transform [10] . Affine and illumination invariant features for color images are presented in [11] ; however, the illumination changes are global and not location dependent.. Most methods employing local features handle illumination changes by normalization of the illumination in the considered area, or by using edge (corner) information, which is less sensitive to the variations in illumination. However, as shown in the following, the localization of the features and the success of the registration process are effected by the changes in lighting conditions.. Global registration methods estimate the parameters of the geometric transformation from the whole image without a prior stage of local feature extraction [12] - [14] . Since global algorithms treat the image as a whole, the background should be separated from the registered part of the image prior to the registration. Global algorithms tend to show robustness to noise. Sensitivity to radiometric changes, on the other hand, is a major disadvantage of most global approaches.. Some affine registration methods avoid dealing with the illumination changes by discarding the radiometric information and treating the image as a binary image. A comparison of several binary registration methods is presented in [15] . It is shown in [15] that although the method complexity grows linearly with the number of pixels, the calculation of moments is only a marginal factor in the total calculation time, mainly due to the need to solve high-order polynomial equations. The reported calculation time is about 1 s even for small images.. The variation in illumination between images of the same object creates a major complication for recognition and registration algorithms. Most registration methods measure properties in the image that are either robust or invariant to the illumination changes [16] . However, such methods assume that the changes in the illumination are location independent; therefore, they are only applicable in small regions of the image.. The dominating approach for representing the effects of location-dependent illumination changes, when no geometric transformation is involved (i.e., camera and object are fixed), is by a linear combination of basis images. The idea of representing illumination changes by a linear combination of images was proposed by Shashua [17] . Hallinan [18] represented illumination changes by a linear combination of basis images and used principal component analysis (PCA) to find an approximation of the basis images. In cases, where the shape of an object is convex and the object is Lambertian (the apparent brightness of the surface is the same regardless of the viewing angle), it was shown by Belheumer and Kriegman [16] that the set of all images under arbitrary illuminations forms a convex cone. The cone can be constructed from as few as three images. Further study was done by Basri and Jacobs [19] , who used spherical harmonics to show that the set of images produced by a Lambertian convex surface lies close to a 9-D space.. As previously mentioned, the difficulties associated with the joint geometric-radiometric estimation problem have led to the current state, where only a few attempts have been made to solve it. The lack of pointwise correspondence (due to the geometric transformation) and the lack of intensitywise alignment (due to the radiometric mapping) do not allow for a simple direct usage of the intensity information of the images. Seemingly, the geometric and radiometric problems are strongly coupled and may not be answered separately. As such, straightforward approaches for solving this problem typically lead to a high-dimensional nonlinear nonconvex optimization problem. Only a few works have explicitly modeled joint geometric-radiometric deformations. Indeed, among these, most evade the inherent nonlinearity of this estimation problem through linear approximation and/or variational optimization-based approaches [20] , [21] . An explicit solution to joint geometric-radiometric estimation problem, where the radiometric change is a nonlinear mapping of the image intensities is given [22] . The assumed radiometric changes, however, are not location dependent. The estimation of location-dependent radiometric changes in the presence of an affine transformation is described in [23] . It requires several images of different spectral bands. Therefore, it is not suitable for grayscale images.. The goal of this paper is to fill the gap between the existing methods for estimating geometric changes only and those for estimating radiometric changes only. We propose a global and linear solution to the problem of estimating the illumination in an invariant manner to the affine geometric changes. Linear, affine invariant constraints on the illumination of an isolated object are derived using image moments. We show that using the derived constraints, the illumination is estimated in an invariant manner to the geometric transformation. The computa- tional complexity of the method is low as it requires no more than solving a linear set of equations.. The paper is organized as follows. Section II defines the problem of jointly estimating the illumination and affine geometric changes. Section III elaborates on the search for linear and affine invariant transformations, and presents a solution to the problem. Section IV introduces linear solutions to the illumination estimation problem. Sections V-VII describe experiments that test the performance and robustness of the illumination estimation, and the effect of prior illumination estimation on the accuracy of registration. Examples of the observations and estimated templates in various poses is displayed in Fig. 13 . In each column, the observation is shown on the top row, while the estimated illumination-compensated template is shown below. Ideally, the difference between the two images is the geometric deformation alone. Table II displays the average error in pixels using MSER and SIFT, with and without prior illumination estimation. The proposed method yields a smaller improvement than in the case of the doll head; however, the prior illumination estimation significantly reduces the number of failed registrations (a registration with an error of more than 200 pixels).

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Combining Invariant Features and the ALV Homing Method for Autonomous Robot Navigation Based on Panoramas Abstract Biologically inspired homing methods, such as the Average Landmark Vector, are an interesting solution for local navigation due to its simplicity. However, usually they require a modification of the environment by placing artificial landmarks in order to work reliably. In this paper we combine the Average Landmark Vector with invariant feature points automatically detected in panoramic images to overcome this limitation. The proposed approach has been evaluated first in simulation and, as promising results are found, also in two data sets of panoramas from real world environments. There is significant research in robotic navigation using methods based on animal navigation techniques. For instance, Carwright and Collet (1983) studied how the honeybees learned and used landmarks to navigate. From this research they created the snapshot model. The idea of this model is to calculate the home vector, which is the vector pointing to the home position. A panoramic image of the target location is created and stored by the animal. Then, when the insect wants to go back to the stored position it uses a matching mechanism to compare the current retinal image to the stored panorama. Another example is the work on robotic navigation of Lambrinos et al (1998 Lambrinos et al ( , 2000 , that took inspiration from the different navigation techniques of the ant species Cataglyphis described by Wehner (1987) . These techniques have the advantage of being computationally cheap. Lambrinos et al (1998 Lambrinos et al ( , 2000 suggest the Average Landmark Vector (ALV) as a way to model these navigation techniques. This model assumes that the animal stores an average landmark vector instead of a snapshot. Landmarks can be (simple) features like edges. The direction to the destination is the difference of the ALV at the destination and the ALV at the current location. The advantages of this model are that it is simple to calculate and that only the orientation and the ALV at the home location have to be stored instead of a whole image. A third advantage is that no matching of the landmarks has to be done.. In robot homing research, artificial landmarks are often used. This is a strong limitation as it requires setting up the environment beforehand. Instead, in our work the goal is to create a simple homing method that can be used without having to rely on artificial landmarks. For this we propose the combination of the ALV homing technique with visual invariant feature detectors, like the ones described by Mikolajczyk et al (2005) , in panoramic images.. More concretely, we propose to use the invariant feature based ALV in conjunction with a global localization method proposed by one of the authors (Ramisa 2006) . The ALV will serve as a costless navigation technique to make the robot go from a starting position to a destination position navigating through a path of place signatures. Another advantage of ALV is that the same visual information as the global localization method of Ramisa (2006) is used, so no additional image processing time is required.. Local homing could also improve the method proposed by Ramisa (2006) in case of ambiguity in the global localization by trying to go to the position in the room where the panorama was made. The most likely hypothesis panorama can be used as a reference for the homing method.. Another contribution of this work is an evaluation of the ALV homing method using feature points extracted from panoramic images of unmodified indoor environments. This is a necessary step towards combining it with the global localization technique of Ramisa (2006) , that would complement the localization method by giving it a computationally efficient method to travel from one node to another of the global map.. Experiments with the ALV homing method were first done in simulation (Goldhoorn et al 2007a,b) and because the results were promising, experiments were also done with real robots (Goldhoorn 2008) in an office environment. Additionally, experiments with artificial landmarks were also done for comparison purposes.. This paper is divided in the following sections: first, some background of the methods used is provided, this includes the panorama acquisition techniques, the local features, the global localization method of Ramisa (2006) and the Average Landmark Vector homing method; next, the proposed method is explained; then the experiments performed to evaluate the method are presented followed by a discussion of the results and finally the conclusions and future work are described. In this work a method to complement the global localization system of Ramisa et al (2008) is proposed, where feature regions computed in a panoramic image were used to characterise a room. The purpose of that method was directing the robot from one of the map nodes to the next with the minimal cost (i.e. no matches have to be established between visual feature points of the images). The method does not rely in any form of artificial landmarks contrarily to previous works, and uses only natural landmarks extracted from the panoramic images acquired with the robot visual system, therefore it can be applied in unprepared environments.. Also, when the robot has several likely hypotheses about its current location (i.e. room) then homing can be used to return to the position where the most likely panorama from the database was made. If the hypothesis with the highest probability was correct, then the panorama at that location should be more similar to the panorama from the database, making it the significantly best hypothesis. In the case where an incorrect panorama was chosen from the database, then the same steps should be taken as before in order to find out in which room the robot actually is.. Although there are several methods to do homing, such as the 1D method of Hong et al (1991) , warping (Franz et al 1998) or snapshots (Lambrinos et al 2000) , the ALV homing method (Lambrinos et al 1998 (Lambrinos et al , 2000 has been used mainly because of its simplicity and low computational complexity.. In order to evaluate the proposed method, initial experiments using a simulated environment were conducted and later it was tested in a real world scenario. The real world experiments were done with panoramas acquired in three different rooms at the IIIA research center.. The locations at which the robot acquired the panoramas were measured manually and used to calculate the ground truth homing directions, which were then used to verify the homing method results. The panoramas were created with the camera on a pan tilt unit which rotates around a fixed point to get images from all different directions. Next, these images were combined to create the panorama. Feature points from these images were extracted to be used by the homing method. Two types of invariant feature detectors were tested: the Difference of Gaussians extrema (DoG) of Lowe (2004) and the Maximally Stable Extremal Regions (MSER) of Matas et al (2002) . Only the horizontal location of the feature points was used, i.e. the cylindrical angle, and not height, nor depth.. The ALV homing was found to be a good working method, however the method performed worse in rooms where the width and length differ greatly. This has been explained by the way the feature points are projected on the panorama and by the equal distance assumption (Franz et al 1998) . Vardy (2005) discusses biologically based homing methods in his thesis and also did experiments with several of them. In his work he used panoramas that were made with a camera pointed to a parabolic mirror. The advantage of creating a panorama like this is the speed of creation. Whereas with our method first images from several angles had to be retrieved and then stitched to create a high resolution panorama. In order to compare both methods of panorama acquisition additional experiments using Vardy's data sets were performed using the SIFT and MSER feature points.. When comparing the results of IIIA data sets and Vardy's data sets we can see that the ALV homing method performs slightly better on the IIIA data sets, but the difference is not significant. There might be several reasons to explain this, such as the difference in resolution, the camera or the environment. For these reasons it cannot be concluded that having a panorama with a higher resolution made by rotating a camera or a camera ring is much better for ALV homing than using a camera and a parabolic mirror. On the other hand, the faster creation of the parabolic panoramas might be more important than the slightly better performance.. Regarding the feature types, in our experiments MSER significantly outperformed SIFT. Also Mikolajczyk et al (2005) confirmed that MSER is one of the most robust feature detectors. The artificial landmarks in the robotics laboratory were used to compare the local feature approach with the more traditional artificial landmarks. As expected, the results with the artificial landmarks were significantly better than with the invariant feature points since they are less affected by occlusions and viewpoint changes. The error with the artificial landmarks was about 7. • less than using the MSER detector (in only the lower half of the panorama). However this difference seems low enough to justify the applicability of the presented homing method, because does not require setting up the environment by placing artificial landmarks. Clearly, the next step is the evaluation of the method in complex navigation experiments combined with Ramisa et al (2008) global localization system. Hafner (2001) mentioned that a magnetic compass does not work very well inside buildings, therefore she used camera information to compensate for that. Extra experiments should be done to verify the stability of the compass orientation (required for the ALV method). Other options to recover orientation from the visual sensors include that of Lambrinos et al (2000) , who used a polarised-light compass which worked good, but needed sunlight from all directions above it and glass windows depolarise the light, therefore it cannot be used inside buildings. Vardy (2005) proposed to use the coherence of flow fields as an indicator of correct orientation, and finally Zeil et al (2003) suggested to use the difference between images to align them by using one-dimensional gradient descent.. For the homing method to work in real-time the panoramas should be created faster and therefore the use of a camera and parabolic mirror is a good option. This however should also be tested with the used global localization method of Ramisa et al (2008) .. Another possible improvement could be using a machine learning method to discard spurious feature points, for example by tracking feature points in a training sequence and modeling those that do not survive long. These feature points are a source of noise for the homing method, and discarding them could improve significantly the results. For the IIIA data set, the type column shows which part of the panorama has been used: all feature points (not filtered), only the feature points at the lower half of the panorama or only at the upper half (see Section 4.2.4). For Vardy data set, the type column shows the number of degrees above and below the horizon of the image which were used. The detector column shows which feature detector has been used to perform homing: DoG, MSER or artificial landmarks which were only available in the robot laboratory. The next three columns: mean, median and std. dev. (standard deviation) show information about the direction error of the home vector in degrees. The calculation of the score is shown in Eqn. 7; 1 being best and 0 being worst. The best home column shows the ID of the location of the home where to the mean error is smallest. Finally the n column shows the number of samples, i.e. different panoramas, for the data set. When calculating the home vector between two points, for example a and b, the home vector from a to b will obviously always point in opposite direction of the home vector from b to a. This means that these are dependent values and therefore only one of them was used in the analysis. Next we discuss the results for the three different areas.. Robotics laboratory: Most panoramas, 38 in total, were acquired in the robotics laboratory, a room of 10.5 m × 11.2 m. Only the half of the room is really used for this experiment because the other part is filled with working places and the robot soccer field as can be seen in Figure 8 .. The home vectors have an error equal to or lower than 90. • in 89.3% of the cases when the DoG detector was used, 92.6% for the MSER detector and 99.6% when the landmarks were used. An error of 10. • or less was obtained in 22.6% of the cases for the DoG detector, 32.7% for the MSER detector and 64.3% for landmarks. Table 1 shows the results for each type of detector used. The homing errors for the three methods are all significantly different (p < 0.001) according to the rank sum test, and the t-test after bootstrapping (n = 1000). From this can be concluded that the homing method worked best with the artificial landmarks, as expected, and worst with the DoG detector.. Square room: The square room is 4.0 m × 3.4 m big. Figure 9 shows a panorama made in this room. Figure 10 shows the map of the room and the home vectors to panorama 137. Table 2 shows the statistics of the homing method using both feature types. MSER feature points achieved lower error rates than DoG feature points, but this is not significant (confirmed by the rank sum test and the t-test) and it must be noted that only three panoramas were created in this room.. Corridor: Although the simulation showed that the ALV homing method works better in square rooms, we wanted to find out what the impact of a very long and very narrow room in a real environment would have on the Table 2 The error of the homing method using the panoramas which were made in the square room. Table 3 The average error of the homing method in the corridor for the different feature types.. method. A corridor was chosen for that reason as last experiment room. The part of the corridor in which the robot moved is 2.2 m wide and about 22.5 m long. In Figure 11 the map of the corridor can be seen. Additionally, Figure 12 shows the panoramas acquired in the corridor.. In Figure 11 the home vectors to panorama 203 are shown. An error of 90. • or less was obtained in 73.3% of the cases for both feature types, an error of 10. • or less was only obtained in one case (6.7%). Table 3 shows the average error of this data set; the differences between the results with DoG and MSER are not statistically significant.. From the results at the different rooms, it can be seen that the ALV homing method worked better in both the square room and the robotics laboratory than in the corridor. This difference might be explained by the previously found conclusion, in the simulated experiment (Section 4.1), that the method works better in approximately square rooms. This is due to the equal distance assumption. In what follows we provide additional details regarding the analysis of the results.. Corridor results: The panoramas acquired in the corridor (Figure 12) show that there are several disturbing factors on which numerous MSER feature points were found. Panorama 198 is the only panorama taken at a corridor intersection, and therefore the MSER detector finds considerable more feature points than in the other panoramas. In panoramas 200 and 201 a door with blinds is visible, and the MSER detector also found a great amount of feature points on these blinds; in panoramas 199 and 200 the robotics laboratory is visible through an open door which again has many feature points. Figures 11.a and 11 .b confirm this, because here the home direction from panoramas 199, 200 and 202 to 203 were good, but from panoramas 198 and 201 really bad. The reason for this is that in panoramas 198 and 201 the most MSER feature points are located on one side only (as commented earlier in this paragraph), while in the other panoramas (199, 200, 202 and the home panorama 203) the feature points are more or less equally distributed. Although only the MSER detector was mentioned here, the DoG detector generated even more feature points, but with a more or less similar distribution.. In Table 4 (see Appendix) can be seen that the best corridor of the IIIA data sets is at rank 25, but this is below the best of the data sets robot lab and square room.. Upper and lower part: In an attempt to improve the results, the view of the image was limited to only the lower half of the panorama. This part contains objects which are closer to the robot and therefore decrease the size of the visible world, for this reason a room may look more square.. In the robotics laboratory using only the lower half of the panorama resulted in a lower error than using all feature points of the panorama (p < 0.001 with the t-test and the rank sum test for both DoG and MSER). For the other rooms there was no significant difference in performance. Also here the best results were when the MSER detector was used (p < 0.005 for the robotics laboratory and corridor ) except for the square room where DoG was the best detector (p < 0.001, rank sum test).. Also the use of only the upper half part of the panorama was tested, but these results were significantly worse than using the whole panorama for the robotics laboratory (p < 0.001, t-test and rank sum test). There was again no significant difference in the square room and corridor.. Depth: When only the position of the feature points on the panorama are used then only the direction of the home vector can be found. However, when the distance to the feature points is available, a more precise estimation of the distance to home can be calculated.. As can be seen from Table 4 (see Appendix) , the scores for Vardy data set vary from 0.85 to 0.3 and the home angle error from 28.2. • ±27.6 to 126.0 • ±43.3. The results are worse than the results with the previously discussed data sets, but it must be noticed that Vardy's data sets contain more samples.. It can be seen that a wider vertical view angle gives better results. When MSER feature points were used, a view angle of 15. • (above and below the horizon) worked significantly better than a lower angle (p < 0.001, t-test and rank sum test). For all data sets except for doorlit and hall1 the best view angle was 20. • . This is also the case when DoG feature points were used, except for the data sets day, hall2 and screen. In the data set day the difference was not significant enough; using a view angle of 5. • had the best results in the sets hall2 (p < 0.001, rank sum test) and screen (p < 0.05, rank sum test) when DoG feature points were used.. It is also clear from Table 4 (see Appendix) that the performance is better when using the MSER detector than the DoG detector. This difference is significant for all data sets with a view of more than 5. • above and below the horizon (using the t-test and rank sum test; p < 0.001). It also can be seen from the table that the best of the IIIA sets are all above the data sets of Vardy, however this is only significant for the robotics laboratory. Comparing the results of Vardy's data set with the results of the IIIA data set is difficult because of several reasons. First of all there are two big differences between them: the environments and the way the panoramas are made. The rooms could be assumed to be quite similar since they both are flat "office like" with several desks, chairs and computers, but the landmarks present in the robotics laboratory are not present in Vardy's rooms for example. To the best of our knowledge no other work has addressed the combination of the ALV homing method with invariant feature points such as the MSER or the DoG.. So far, in most works that studied the ALV homing method, artificial landmarks have been used. For example Lambrinos et al (2000) used as landmarks four black vertical cylinders, and in (Möller 2000) experiments were done inside of a white box with several wide black vertical stripes on the walls. Möller et al (2001) did extensive experiments in a desert type outdoor scenario with four black cylinders as landmarks. In this same work an experiment was attempted in an indoor scenario. Natural landmarks where found by vertically averaging a certain area of the image and finding edges (i.e. intensity jumps) in the unidimensional graylevel profile.. Hafner and Moller (2001) investigated if a MultiLayer Perceptron with backpropagation and a Perceptron with Delta Rule were able to learn a homing strategy both in simulation and in real world experiments. For the real-world experiments panoramic images acquired by the robot camera were reduced to a single line by vertically averaging (similarly to what Möller et al (2001) did), thus the input of the neural networks is a unidimensional image. Both neural networks successfully learned a homing strategy with the same characteristics as ALV. Usher et al (2003) used a version of ALV augmented with depth information to guide a car-like vehicle in an outdoor experiment. Landmarks were salient color blobs and the depth information was acquired directly from the distance of the landmark to the center of the omnidirectional image (no unwrapping is performed) using a flat-world assumption. The authors performed real-world experiments using red traffic cones (witch hat model) as landmarks.. Vardy (2005) did an extensive study for a variety of biologically plausible visual homing methods in his PhD thesis, both for local and associative methods, in a real office environment. Among the methods evaluated in his work, there is the one proposed in Hafner and Moller (2001) , referred to as Center of Mass ALV. In the experiments it performed similarly to other local homing methods, although it was found that an extra learning phase was necessary to determine which area of the panoramic image should be used to generate the unidimensional image in certain environments. In this section we describe the different concepts and techniques that have been used in this work. In these experiments several panorama were acquired at a grid of known points in the rooms. The orientation of the robot was kept constant for each panorama so no alignment step is necessary between the panoramas.. Three types of landmarks/feature points were used: 1) DoG feature points; 2) MSER feature points; and, only in the robot laboratory, 3) artificial landmarks. The experiments were done in three rooms of different sizes: the robot laboratory , the square room and the corridor. A scaled map of the rooms can be seen in Figures 8, 10 and 11 .. The locations where the panoramas were created are marked as circles with its identifying number and a line starting at the center of the circle and pointing to the direction of the estimated home vector. The home location is shown as a red circle without line and is also indicated in the figures captions. The biggest objects in the rooms, such as desks, are also shown in the maps to give a rough idea of the environment. Finally, the squares in Figure 8 show the landmarks positions and its ID number.. Like in the simulation, only the direction of a feature is known and not its distance, therefore the home vector will not contain distance information either. The home angle calculated by the homing method is compared to the ground truth home angle which is calculated by geometry.. All angles are in degrees and counter-clockwise; h c is the correct homing direction calculated by using the positions (geometry), and h h is computed by the homing method. To find out how well the method works for each room and each type of feature, all the panorama positions per data set are used. For each data set (the square room, the robot laboratory and the corridor ) all the locations where a panorama was created are used to calculate the home vector to each of the other locations. From the error calculated with Eqn. 6 for each possible panorama pairings in one room, the mean, median, standard deviation and a score are calculated. The score is calculated by using the proportion of the maximum error and ranges between 0 and 1 where 1 is best. Namely:. where n is the number of panoramas in the set and P the set of panoramas. The numerator is the sum of the difference of the home angle calculated by the ALV homing method and by geometry. This difference, i.e. error, is calculated for each panorama pair, which in total are n(n − 1) pairs. The sum of errors is divided by that factor to get an average and, to normalise the score between 0 and 1, it is also divided by the maximum possible error, which is 180 • .

O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Hardware/software partitioning using integer programming Abstract Embedded systems typically consist of application specific hardware parts and programmable parts, i.e., processors like DSPs, core processors or ASIPs. In comparison to the hardware parts, the software parts can be developed and modified much easier. Thus, software is less expensive in terms of costs and development time. Hardware however, provides better performance. For this reason, a system designer's goal is a system which fulfills all performance constraints by using as few as possible hardware. Hardware/software codesign deals with the problem of designing embedded systems, where automatic partitioning is one key issue. This paper describes a new approach in hardware/software partitioning for multiprocessor systems working fully automatic. The approach is based on integer programming (IP) to solve the partitioning problem optimally. A formulation of the IP-model will be introduced in detail. The drawback of solving IP-models often is a high computation time. To reduce the computation time, a second approach has been developed which splits the partitioning approach in two phases. In a first phase, a mapping of nodes to hardware or software is calculated by estimating the schedule times for each node with heuristics. During the second phase a correct schedule is calculated for the resulting HW/SW-mapping of the first phase. It will be shown that this heuristic scheduling approach strongly reduces the computation time while the results are nearly optimal for the chosen objective function.. Another new feature of our approach is the cost estimation technique. The cost model is not calculated by estimators like other approaches, because the quality of estimations is often bad and estimators do not concern compiler effects. In our approach the tools (a compiler for the software parts and a high-level synthesis tool for the hardware parts) are used instead of special estimators. The disadvantage of an increased runtime for calculating the cost metrics is compensated by a better quality of the cost metrics compared to the results of estimators. Furthermore, better cost metrics lead to fewer partitioning iterations.. The outline of the paper is as follows: Section 2 gives an overview of related work in the field of hardwarelsoftware partitioning. In section 3 our own approach to partitioning is presented. A formulation of the hardware/software partitioning problem follows in section 4. Section 5 describes the problem by an IPmodel. After experimental results of solving these IPmodels have been presented in section 6, a conclusion is given in section 7 . This paper presents a new approach of full-automated hardware/software partitioning supporting multiprocessor systems, interfacing and hardware sharing. The partitioning approach itself is based on integer programming leading to optimal results. In contrast to other approaches, where hardware and software costs are estimated, our approach follows the idea of 'using the tools' for cost estimation. The disadvantage of an increased calculation time is compensated by better metrics and therefore fewer iteration steps. The presented results are very promising, because nearly optimal results are calculated in short time. Future work will deal with design studies of real system level examples. The interesting parameter for partitioning is the number of nodes n which have to be partitioned. For this reason, we have developed some examples containing a lot of instances of small VHDL-entities. The target architecture for all examples consists of a processor, an ASIC, memory and a bus connecting all components. All calculated partitionings concern interface costs and sharing effects between nodes. The computation times of the examples represent CPU seconds on a Sun SPARCstation20. The heuristic partitioning approach can be evaluated by examining 0 the quality and 0 the computation time compared to the optimal result.. The quality of the heuristic approach can be evaluated by determining the deviation between the exact and the approximated solution. If the heuristic partitioning approach does not consider interfacing, then the results are always exact, and therefore optimal. If interfacing is considered however, then the approximated system execution time may differ from the exact value. Therefore, we have partitioned 6 different systems (see figures 6,s) with the optimal and the heuristic approach. For each system, solutions have been calculated for a set of constraints. In figure 6 it is. . . .--n=7 (heuristic) 7 (optimal) n=4 (optimal). Figure 6: System execution time (exact/heuristic approach). shown that the approximated execution time is equal or very close to the exact value. The maximal deviation between the exact and the approximated execution time is 5.13%, the average deviation is smaller than 1% for all examined systems. In contrast to the partitioning quality, the computation times are very different. Figure 7 depicts the computation times of both approaches for a system, which consists of 7 nodes. This system has been partitioned for 8 different system execution time constraints. The maximal computation time is 246 seconds for the optimal partitioning approach and 2 seconds for the heuristic one, i.e., computation time is drastically decreased. In figure 8 the computation time of the heuristic approach is depicted for 6 different systems. For all of these systems several different There are only few approaches considering hardware/software partitioning.. One of these is the COSYMA system [EHB93] , where hardware/software partitioning is based on simulated annealing using estimated costs.

B I O B I O O B O O O O O B O O O O O O O O O O O O O O O O B O O B O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B B I O O O O O O O B O O O O O O O O B O O O O O B I O O B O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O B O O O O B O O O B O O O O O O O O O O O O O O O O B O O O O O O B O O B O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B I O O B O O B B O O O O O O O B I O O O O O O O O O O B O B O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O