Analysis of a spatial gene expression database for sea anemone Nematostella vectensis during early development Background: The spatial distribution of many genes has been visualized during the embryonic development in the starlet sea anemone Nematostella vectensis in the last decade. In situ hybridization images are available in the Kahi Kai gene expression database, and a method has been developed to quantify spatial gene expression patterns of N. vectensis. In this paper, gene expression quantification is performed on a wide range of gene expression patterns from this database and descriptions of observed expression domains are stored in a separate database for further analysis. Methods: Spatial gene expression from suitable in situ hybridization images has been quantified with the GenExp program. A correlation analysis has been performed on the resulting numerical gene expression profiles for each stage. Based on the correlated clusters of spatial gene expression and detailed descriptions of gene expression domains, various mechanisms for developmental gene expression are proposed. Our cluster analysis indicates that early gene expression domains in N. vectensis are spatially separated in a stable sequence along the primary body axis. An additional statistical analysis indicates that precise gene expression domains in N. vectensis are generally formed by two processes. Genes that are expressed in the blastula appear in broad expression regions. During gastrulation and planula development, the expression domains are refined within the boundaries of these broad regions.. It should be stressed that no additional experiments have been performed for our study. Spatial expression data in a public gene expression database have been quantified and analysed, and these analyses resulted in new hypotheses on cellular migration in N. vectensis. Spatial gene expression patterns have been collected in extensive databases for other model animals as well, and a similar computational approach can generate hypotheses about cellular migration before fate maps are available for these animals. A gene with expression profiles from multiple samples at one stage can belong to more than one cluster. This may be due to the noise in gene expression among individuals. Another possible explanation is that the expression patterns are different across individual in situs at the beginning and at the end of a developmental stage. This change could be caused by cells migrating from one region to another, or by dynamic regulatory interactions. In the current study, this issue is handled by performing cluster analyses to all expression profiles for all genes within a broad time window instead of analyzing blurred averages of each gene. A solution to this uncertainty would be an increased time resolution for the expression profiles, resulting in more precise regions for narrower time windows. In Drosophila embryos, the definition of narrow time classes allowed the observation of significant domain shifts [4] . In the sea urchin, precise timing resulted in a sequence of spatial regulatory states [16] .. Central domain expression is generally persistent in the endoderm, while expression in the central and external rings is often limited to the oral ectoderm during and after gastrulation. The first entry in Table 1 shows that the early gastrula expression pattern is known for 23 genes that are expressed exclusively in the central domain in the blastula stage. Out of these 23 genes, 21 are expressed in the Table 2 . Moreover, expression is observed exclusively in the same major domain in the next available stage for at least 69 % of all instances of expression limited to either of these two major domains, as indicated in the last column of Tables 4 and 5 . These observations are a strong indication that the central domain differentiates into the later endoderm and that the central and external rings become oral ectoderm. The vegetal hemisphere likely becomes aboral ectoderm, but this is based on a small number of expression patterns (Table 6 ). These differentiation regions are in agreement with the locations of a dye injected into an N. vectensis egg and recorded during embryonic development [11] . Injection of lineage tracers into individual cells of developing N. vectensis embryos results in contiguous clones of labeled cells without long distant migration of individual cells (Martindale, unpublished observations). Based on the statistical stability of the main gene expression regions, a regulation mechanism is proposed where genes in early regions activate genes in corresponding later regions, while genes in adjacent regions repress one another (Fig. 21) . If changes in expression patterns occur consistently in many genes, this might indicate a collective cellular motion in the embryo. During gastrulation, expression in the region between the oral and vegetal poles is observed for the quantified profiles of Anthox1, FGFRa, FoxD.1, Sox1, Sox3 and Rx. Due to intense Sox1 expression in the oral pole, the Sox1 profiles are outside the vegetal ring clusters. Anthox1, FGFRa and FoxD.1 are likewise classified as members of the vegetal pole cluster because their strongest expression occurs in the corresponding region. For these six genes, a For each period, the possible changes in gene expression (minor change:major change:vanished) are added in the second column and expressed as percentages in the third column. As an example, the period from the early gastrula to the mid gastrula (bold numbers) is included in the combinations of stages highlighted in Table 1 Table 5 This hybridization experiment is a weak indication that genes in the body wall ectoderm are first expressed in the external ring or vegetal ring of the blastula. This is basically the null hypothesis; in the absence of any known collective cell movements in the aboral ectoderm during gastrulation, expression in this region has likely remained stationary from the blastula. The body wall has been included in the proposed regulatory interactions among clustered gene expression regions (Fig. 21) . The vegetal domains of Anthox1, FGFRa and FoxD.1 become restricted to the aboral end in the planula stage, while Sox1 and Sox3 move towards the oral pole in the planula. Meanwhile, the Rx domain has significantly expanded in the late planula. These changes in gene expression domains may indicate that ectoderm elongation in the planula stage is most pronounced in the initially narrow Rx expression domain. Even though this hypothesis is based on few observations, it could be tested with fate mapping experiments. In the frog Xenopus laevis, the gene rax promotes cell proliferation in developing retinal tissue [17] ; Rx may similarly induce tissue growth in N. vectensis. Many hierarchical clusters include profiles with strong pairwise correlations to profiles in other clusters. This is caused by the partial overlap of the expression domains that characterize the clusters and by expression of some genes in both regions. The average cutoff value for the main clusters is 0.9; this value is quite high and indicates fuzzy boundaries between many expression clusters. These fuzzy boundaries between expression clusters are likely due to dynamic changes in gene expression. Out of the 25 genes expressed only in the central domain, 3 (12 %) will exhibit major expression changes, or terminate their expression (Table 4 , first row). Out of the 10 genes expressed in the central ring or external ring, 1 (10 %) is not expressed in the oral ectoderm (Table 5 , first row). This explains why some genes are expressed in multiple clusters at the same stage, and why some genes appear in clusters for different expression domains between developmental stages.. According to the hypothesis that the position of cells in the planula is largely determined by their position in the blastula, the change of gene expression patterns across major regions must be the result of regulatory action. The added percentages for possible major change and first appearance of gene expression in the early gastrula to early planula stages are 45, 48 and 55 % (Table 8 ). In comparison, these percentages are 37 and 25 % in the early planula to late planula stages. These observations suggest that more dynamic changes in expression occur during gastrulation than in the period that the planula transforms into a polyp. For the study of pattern formation in N. vectensis, recording gene expression during the relatively short period of gastrulation should therefore be more informative than monitoring the planula stages. In general, during gastrulation it is decided in which major region(s) the genes are expressed. Detailed expression patterns arise during the planula stages, generally within the bounds of the major regions determined for each gene at the end of gastrulation. The appearance of differential details explains the decrease in correlation among gene expression patterns after gastrulation (Figs. 10, 11, 12, 17, 18 and 19 ). This loss of correlation could also be caused by the lower number of quantified patterns in the planula stages; with additional gene expression quantifications in these stages of development it could be tested whether the expression domains diverge or whether they form a new set of clusters.. One aspect of gene expression that requires spatial localization, is the determination of the secondary axis in N. vectensis. Individual cells appear indistinguishable during the early cleavage cycles until the early blastula starts oscillating at the animal pole [18] . This oscillation stops at the mid blastula transition, when the synchrony of cell divisions is lost and the blastula remains spherical. The spherical blastula symmetry is permanently broken at the onset of gastrulation [19] . During gastrulation, Bmp ligands and antagonists are asymmetrically expressed along the secondary axis [13] . Based on these observations, determination of the secondary axis may coincide with the mid blastula transition. The asynchrony in cell divisions then produces a stochastic perturbation in a morphogen concentration which would define the secondary axis.. The gene expression database contains various genes that are differentially expressed along the secondary axis, although most are not identified in the expression summaries. From the incomplete list of genes included in the expression pattern overview in this study (Additional file 2), various genes involved in secondary axis formation can already be identified. The genes Anthox7, Anthox8b, Bmp2/4, chordin, Gbx, Hex, Msx, Msx2, NvHD060, Smad1/5 and Vent1 exhibit noncylindrical expression patterns. Quantification of these patterns requires a two-dimensional or three-dimensional template and eventually a three-dimensional detection method. Especially Vent1 asymmetric expression in the early gastrula stands out. In various animals, vent genes are involved in a Bmp signaling feedback loop [20] . Curiously, the genes Bmp2/4, chordin and Smad1/5 in this signaling pathway are still symmetrically expressed in the N. vectensis early gastrula (or at least their asymmetry is less pronounced).. The N. vectensis in situ hybridization collection contains expression data for less than half of all developmental stages for most genes stored. Despite the sparsity of this dataset, a large-scale analysis results in meaningful insights. Sparse data is common for databases that contain labor intensive measurements, and standardization allows global analysis approaches to be applied to incomplete biological databases. Developmental gene expression databases contain large sets of genes with many regulatory interactions. Our clustering-by-region approach is convenient to select genes from a large set for computational regulatory network modeling. Spatial gene expression assays can be used as a tool for verifying predicted regulatory interactions between genes and for predicting properties of missing components in a gene regulation network [1, 2] . The largest potential of spatial gene product distribution datasets, is in verifying numerical models of regulatory interaction networks, which has been demonstrated for the embryonic development of fruit fly Drosophila melanogaster [3] . Also the formation of digits in early mouse limbs has been replicated with mechanistic models with the help of gene expression patterns [4] .. To perform accurate simulations of such processes, the spatial gene expression patterns need to be digitally quantified and formatted to standardized profiles. A procedure for gene expression quantification has been described [5] and applied [6] for organisms with changing morphologies during embryonic development. Similarities among gene expression profiles can provide information about co-expression relationships [7] . Similarity metrics are a common tool for classifying time series expression data to identify correlating dynamics among genes. These similarity measures can identify correlating spatial expression among genes from quantified expression patterns as well. To use quantified gene expression patterns in dynamic simulations, reliable time points for gene expression patterns are required.. For example, in Drosophila, the spatial gene expression of even skipped (eve) has been measured precisely for many time points. The eve pattern is employed as a time reference: eve is assayed together with the queried gene to establish the development time for the sample [8] . For many other animal models, a time reference gene is not (yet) available and other embryo properties are applied to estimate the time of development. In these cases the subsequent stages of development can be qualitatively identified from the changing embryo morphology. These changes in morphology are caused by division and migration of cells, processes that are absent during the early cleavage cycles of flies.. In the comparative gene expression database Kahi Kai, in situ RNA hybridization assays are collected for many marine invertebrates [9] and are classified according to the embryo morphologies. This database thereby allows for an analysis of spatial expression features for all gene entries.. In this study, many genes from the Kahi Kai database are compared at various stages of development, based on their expression in different embryonic regions. First, the majority of in situ hybridization images are quantified and Table entries indicate the hours after fertilization derived from [11, 18, 21] . The annotations in the schematic morphologies are guidelines for researchers to describe expression domains in their hybridization images. AnHe = animal hemisphere, VeHe = vegetal hemisphere, An = animal pole, Cd = central domain, Cr = central ring, Er = external ring, Ve = vegetal pole, pEn = presumptive endoderm, bEc = blastoporal ectoderm, Ec = ectoderm, OrHe = oral hemisphere, AbHe = aboral hemisphere, OrEc = oral ectoderm, En = endoderm, AbEc = aboral ectoderm, PhEc = pharyngeal ectoderm, PhEn = pharyngeal endoderm, AtEn = apical tuft endoderm, AtEc = apical tuft ectoderm, At = apical tuft, M = mouth, BwEc = body wall ectoderm, BwEn = body wall endoderm, MeEc = mesentery ectoderm, MeEn = mesentery endoderm, TeB = tentacle bud, TeEc = tentacle ectoderm, TeEn = tentacle endoderm, Si = siphonoglyph, TeTi = tentacle tip, TeBa = tentacle base (The original nomenclature in [9] has been adapted to the more detailed denotations for the blastula stage in [15] .) the quantified gene expression patterns are collected in a list of digital expression profiles. Stage-specific correlation analyses are performed on these spatial gene expression profiles to discover the embryo's major division in expression domains.. Second, a subset of genes from the database is listed with a detailed description of the spatial expression in the stages for which data is available. This list provides an overview of the developmental stages with spatial hybridization images for each gene and allows a detailed description of expression properties beyond the general classifications. Progression of spatial expression is compared for subsequent available stages, and the main periods of gene expression dynamics are identified.. A large set of gene expression patterns in the starlet sea anemone Nematostella vectensis is analyzed. The determination of the secondary axis in N. vectensis is one aspect of gene expression that requires spatial localization. The database contains various genes that are expressed along this axis.. The change in N. vectensis morphology during development is schematically displayed in Fig. 1 . The nucleus in the egg is located at the future oral pole, which means that the primary (oral-aboral) axis is already determined before fertilization [10, 11] . The determination of the secondary axis, which is defined by the location of the syphonoglyph, is unclear. The first structures that appear along this axis are the primary mesenteries, but differential gene expression is already observed during gastrulation [12, 13] . Based on the early symmetry break in various gene Fig. 3 Gene expression quantification pipeline. A digital morphology is overlaid with the gene expression image (a) and the points are manually dragged over the embryo's cell layer (b). After the cell layer is decomposed into segments (c) and the intensity is plotted as a function of cell layer position (d), the profile is manually edited to correct for artifacts (e) Fig. 2 Workflow overview. The information stored in the Kahi Kai gene expression database has been processed into convenient formats for two partly overlapping sets of genes. These processing methods and the methods used for additional analyses are described in the text. While this workflow may seem to converge to a single final result, all intermediate results can be explored for multiple purposes Fig. 4) . Expression in the presumptive endoderm at the early gastrula is incorrect expression patterns and on early N. vectensis morphogenesis, a mechanism is proposed for secondary axis determination. Spatial gene expression patterns in early stages of development are necessary to study the determination and formation of the secondary axis.. We analyzed spatial gene expression patterns in various stages of development in N. vectensis. Changes in these patterns are due to gene expression dynamics within stationary cells or due to migrating cells that retain their gene expression state. Fate mapping experiments can conclusively determine migratory behavior, but these data are not available for N. vectensis. Our solution to deal with this lack of fate map data is the assumption that the expression state in migrating cells likely remains unchanged for many genes. In this fashion, we estimate major cell movements based on the spatial gene expression data, which are available.. In conclusion we demonstrate the application of correlation analysis to quantified spatial gene expression patterns in order to identify co-expressed spatial domains.. A possible application of these correlation matrices is the selection of gene clusters for regulatory experiments, functional studies [14] and computational gene regulation network models [6] , because co-expressed genes often share regulators or biological functions. The order of application for the described methods is displayed in the diagram in Fig. 2 . Note that the intermediate results provide new information on their own and can be subjected to additional analyses.

O O O O B I I O O O B I O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


HouSI: Heuristic for delimitation of housing submarkets and price homogeneous areas This paper seeks to address the problem of the empirical identification of housing market segmentation, once we assume that submarkets exist. The typical difficulty in identifying housing submarkets when dealing with many locations is the vast number of potential solutions and, in such cases, the use of the Chow test for hedonic functions is not a practical solution. Here, we solve this problem by undertaking an identification process with a heuristic for spatially constrained clustering, the ''Housing Submarket Identifier'' (HouSI). The solution is applied to the housing market in the city of Barcelona (Spain), where we estimate a hedonic model for fifty thousand dwellings aggregated into ten groups. In order to determine the utility of the procedure we seek to verify whether the final solution provided by the heuristic is comparable with the division of the city into ten administrative districts. A housing market is the collection of alternative locations considered by households as location substitutes, with houses in different submarkets being imperfect substitutes within the same broader market.. 1 Housing market segmentation is widely recognised in the literature, and several studies have stressed its importance. Goodman and Thibodeau (2007) propose a number of reasons as to why an understanding of how metropolitan areas are partitioned into housing submarkets is important: (a) it increases the accuracy of statistical models when estimating house prices (Goodman & Thibodeau, 2003) ; (b) it improves the modelling of spatial and temporal variations in house prices; (c) it improves the ability to price the risk associated with financing homeownership for lenders and investors; (d) it can reduce search costs for those demanding housing; and (e) it avoids inducing spatially correlated errors that bias the coefficients on variables correlated with the errors. Bourassa, Cantoni, and Hoesli (2007) further emphasise the importance of submarket differentiation in their study of house price prediction in a mass appraisal context. They report equally, or even more, accurate predictions when using a traditional hedonic model incorporating a series of dummy variables to define submarkets as when using more complicated models (such as lattice SAR and CAR or geostatistical models). Many techniques have been adopted in the detection of submarkets and housing segmentation: cluster analysis (Bourassa, Hamelink, Hoesli, & MacGregor, 1999) , GIS and ESDA analysis (Tu, Sun, & Yu, 2007) , cointegration analysis (Jones et al., 2003) , fuzzy clustering (Hwang & Thill, 2009 ), non-parametric smoothing and spline functions (Pavlov, 2000) , neural networks (Kauko, 2004) , classification regression trees (Fan, Ong, & Koh, 2006) , and household mobility patterns (Jones, Leishman, & Watkins, 2004; Royuela & Vargas, 2009 ). The most frequently applied technique by far has been the use of hedonic models for house prices. The basic assumption is that hedonic coefficients for housing characteristics, such as living space, capitalise neighbourhood amenities, such as public education. Separate models for global housing markets and for housing submarkets are computed and, then, F-tests for nested models are used. These tests tell us whether or not there is a significant reduction in the sum of squared residuals by splitting the subsample into submarkets. The usual result is that housing submarkets matter and this is what is to be expected, especially when using large datasets. In this study, we adopt this approach, considering as our crucial criterion the proximity between the vectors of hedonic price characteristics of each submarket.. The main problem faced by researchers is how best to combine a large number of spatial units into a smaller number of housing submarkets, considering that the number of different ways that N dwellings or areas can be grouped into M submarkets is particularly large (Cliff & Hagget, 1970; Cliff, Haggett, Ord, Bassett, & Davies, 1975; Keane, 1975) . 2 The literature has tackled this problem in two ways. On the one hand, different methods have been used to delineate submarkets (see above). On the other hand, the literature employs a small number of previously defined (in administrative terms, for instance) submarket regions, which might include districts or census tracks (as in Watkins, 2001 , 3 Adair, Berry, & McGreal, 1996; Goodman & Kawai, 1982; Schnare & Struyk, 1976) . Both approaches provide discrete submarkets, which has been criticised on the grounds that neighbourhoods (housing submarkets) are not easily defined a priori and, consequently, continuous solutions are deemed preferable (Redfearn, 2009; Sunding & Swoboda, 2010) . Our paper seeks to address the problem of the empirical identification of housing market segmentation, once we assume that submarkets exist. We undertake this identification process by implementing a heuristic for spatially constrained clusters, the ''Housing Submarket Identifier'' (HouSI). This heuristic combines two strategies: (a) the construction of initial feasible solutions based on the region-growing strategy (Taylor, 1973 and Openshaw, 1977) , and (b) the local improvement of a feasible solution based on the Tabu Search algorithm (Openshaw & Rao, 1995 and Duque, Anselin, & Rey, 2011) . A more formal definition of the problem of aggregating N areas into M regions, while optimising a predefined aggregation criterion, is available in Duque, Ramos, and Surinach (2007) and Duque, Church, and Middleton (2011) .. HouSI builds on the existing literature on modelling housing submarkets. It achieves this by using estimates of hedonic parameters and their standard errors, which are then incorporated into the objective function to assess the solution quality. Moreover, HouSI can improve the use of hierarchical linear modelling (Goodman & Thibodeau, 1998) as its results depend neither on the starting point nor on the order of adjacency.. It is our belief that the use of discrete housing submarkets rather than continuous delineations of housing submarket differences (built for instance with local weighted regressions) 4 affords the following advantages: (a) it helps policy makers to better define place-based policies; (b) it improves political decisions by targeting the right space to be controlled (Jenkins, 1978) and by ensuring that the specific public policy actions implemented in a region have a homogeneous impact throughout that region (Fischer, 1980) ; (c) it is easy to implement and understand; and (d) in mass appraisal contexts, the consumption of degrees of freedom is not a great problem. We examine housing market segmentation by conducting a case study in the city of Barcelona (Spain), where we build a small number of submarkets starting from 40 postal districts. As Hwang and Thill (2009) argue, the body of literature has leaned toward testing the distinctiveness of housing submarkets given a priori housing submarkets. Consequently, in order to determine the utility of the procedure we seek to verify whether the final solution provided by the heuristic is comparable with the division of the city into ten administrative districts (as in Adair et al., 1996; Goodman & Kawai, 1982; Schnare & Struyk, 1976) . Our results show that the final division of the city undertaken with HouSI is superior to that of the administrative districts identified in terms of housing prices by the Chow test, and that it is statistically significantly better than any random aggregation of housing submarkets.. For simplicity's sake, we do not undertake a review of the literature (for a survey of housing submarkets and related issues, see Watkins, 2001; Kauko, 2004; Goodman and Thibodeau, 2007; Páez, 2009; and Islam & Asami, 2009 ), but focus rather on presenting the heuristic (Section 2), the housing price hedonic models and case study (Section 3), the results (Section 4), and the main conclusions of our work (Section 5). This paper's prime aim has been to explore new alternatives for constructing housing submarkets when merging large numbers of spatial units into a small number of such submarkets. Housing submarkets are characterised by their wide range of hedonic function parameters and, thus, two neighbourhoods can be said to belong to the same housing submarket when their hedonic functions are similar. Here, we have proposed a spatially constrained clustering heuristic, HouSI, specifically designed for identifying housing submarkets. HouSI performs parameter comparisons between polygons using a metric built as a Wald test of structural change for two subsamples of different variance plus a generalised Wald test for comparing a list of parameter vectors.. The empirical evidence indicates that HouSI serves as a potentially good alternative to the massive use of Chow tests. Future research should be directed towards endogenizing the parameter M, so that the optimal number of submarkets can be defined by the heuristic. We estimated a log-log function for our hedonic model in which the structural characteristics (age and size) were combined with a list of time dummies. Subsequently, we incorporated dummies for the city's postal districts and the interactions with the structural variables of age and size. In order to identify the model's sources of explanatory power, we considered starting with a simple model of structural characteristics (Model 1), subsequently expanded to include time dummies (T t , in Model 2), local dummies (D i , in Model 3), and the interactions of these dummies with the structural characteristics (Model 4).. Model 2 : lnðpÞ. 15 Note that at each iteration, the best neighboring solution does not necessarily lead to an improvement of the aggregation criterion (also known as the current solution); however, non-improving moves of this type are allowed for a limited number of consecutive iterations to give the algorithm the capacity to escape from local optimal solutions. 16 Although the Tabu Search algorithm has been widely applied within the context of spatially constrained clustering, there are only a few studies that evaluate the capacity of the algorithm to navigate the solution space for different parameter values (Blais, Lapierre, & Laporte, 2003; Bozkaya, Erkut, & Laporte, 2003; Ricca & Simeone, 2008) . Many papers, including those by Openshaw, do not even mention the number of iterations defined as stopping rule. In our case we decide to use the highest value for the stopping rule reported in the peer reviewed papers that apply this algorithm for spatial clustering. Bozkaya et al. (2003) use 115sqrt(m) and 230sqrt(m); Ricca and Simeone (2008) use 280; and Blais et al. (2003) use 100sqrt(m) ; where m is the number of regions. As pointed out by Bozkaya et al. (2003) , using sqrt(m) instead of m is a common practice that allows us to take into account the problem size in the parameter settings. Since the higher the parameter value for the stopping rule, the higher the possibility of getting a better solution, we chose to use the highest reported value for this parameter: 230sqrt(m). 17 This approach has been widely used to estimate the value of green areas (Gunn, 2007) , forests (Hand, Thacher, & McCollum, 2008) , improvements in transportation systems (Yiu & Wong, 2005) , public goods (Gravel, Michelangeli, & Trannoy, 2006) , among others. 18 When researchers have spatially disaggregated data, the use of locational variables for every dwelling can be a much more appropriate procedure. 19 The database employs statistics provided by appraisal firms, as the real prices of housing transactions are not published in Spain. We are aware that our procedure presents certain weaknesses, including problems of bias (Dietrich, Harris, & MullerIII, 2000) ; dispersion (Hansz and Diaz-III, 2003) ; and econometrics (see Bond & Hwang, 2007 , for a list of such problems associated with appraisals). However, our main concern is whether the appraisal firms use a territorially-biased method or procedure in their computations. Fortunately, we understand that this is not the case in Barcelona, where appraisal firms are large and officially accredited. Thus, we assume that price adjustments are not persistently inconsistent in space. This database was previously used in Royuela and Vargas (2009) . 20 We would have worked with a broader final database if we had had access to more structural characteristics. Regrettably, this was not the case.. V. Royuela, J.C. Duque / Computers, Environment and Urban Systems xxx (2012) xxx-xxx. Model4 : Table 1 shows the basic results obtained with these models. The structural characteristics explain more than 50% of the model's total variance (model 1). We find that doubling the size of the dwelling involves more than doubling its price, while doubling a house's age results in a 5% fall in price. Adding time dummies (model 2) explains an additional 25% of the total variance, while incorporating spatial differentiation via the 40 postal districts (model 3) helps to explain an additional 10% of the total variance. As suggested by Bourassa et al. (2007) , all estimates were performed using simple OLS regressions.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


A unified approach for detecting and eliminating selfish nodes in MANETs using TBUT Recent years have witnessed the increasing efforts toward making architecture standardization for the secured wireless mobile ad hoc networks. In this scenario when a node actively utilizes the other node resources for communicating and refuses to help other nodes in their transmission or reception of data, it is called a selfish node. As the entire mobile ad hoc network (MANETs) depends on cooperation from neighboring nodes, it is very important to detect and eliminate selfish nodes from being part of the network. In this paper, token-based umpiring technique (TBUT) is proposed, where every node needs a token to participate in the network and the neighboring nodes act as umpire. This proposed TBUT is found to be very efficient with a reduced detection time and less overhead. The security analysis and experimental results have shown that TBUT is feasible for enhancing the security and network performance of real applications. Recent years have witnessed the increasing efforts toward making architecture standardization for the secured wireless mobile ad hoc networks. Actually, the framework of secured mobile ad hoc networks (MANETs) is an important part of the next-generation network design [1] [2] [3] [4] . Such an improved security network brings a bright foreground for large data communications, i.e., the demand for the large data applications like IPTV, VoIP, and video conference has grown tremendously [5] [6] [7] [8] . Meanwhile, research topic on secured communications has also received much attention in the past decade, and many works have been proposed to design robust and efficient schemes for delivering secured content delivery over error-prone networks [1, 5, 6, [9] [10] [11] [12] [13] .. This paper is based on the foundations of two systems proposed by Kathirvel and Srinivasan, namely selfumpiring system (SUS) [5, 14] and enhanced triple umpiring system (ETUS) [1, 5] . In the self-umpiring system, each node is issued with a token at the inception. The token consists of two fields: NodeID and status [5] . NodeID is assumed to be unique and deemed to be beyond manipulation and status is a single-bit flag. Initially, the status bit is preset to zero, indicating a green flag. The token with a green flag is a permit issued to each node, which confers it the freedom to participate in all network activities. Each node in order to participate in any network activity, say, Route Request (RREQ), has to announce its token. If its status bit is "1" indicating a "red flag," the protocol does not allow the node to participate in any network activity. The working of the self-umpiring system is explained with reference to Figure 1 .. In the self-umpiring system, all the nodes have dual roles -packet forwarding and umpiring. In the forward path during data forwarding, each node monitors the performance of its immediate next node. That way, mobile node A can tell correctly whether mobile node B is forwarding the packet sent by it, by promiscuously hearing mobile node B's transmissions. Similarly during reply process RREP, mobile node C can verify whether mobile node B is unicasting the Route Reply (RREP) and whether the hop count given by mobile node B is correct. Thus, during forward path, mobile node A is the umpire for mobile node B and mobile node C is the umpire for mobile node B during reverse path operations. When a node is found to be misbehaving, say dropping data packets, the corresponding umpire immediately changes the status bit of guilty node to "1" indicating a red flag.. The ETUS model [1, 5] is presented in Figure 2 . The active path is specified by node source, node 1, . . node N i -1 , node N i . . node N m , and the destination node. Thus, there are N m +2 nodes in the active path U 1 , U 2 . . U i , U i + 1 . . U m and U m + 1 are umpiring nodes. Umpire U i is situated in the communication zones of nodes N i , N i − 1 , U i − 1 , and U i + 1 . For node N i , the two umpires will be U i and U i + 1 . The third umpire will be N i − 1 in the forward path and N i + 1 in their reverse path. Thus, when N i is found to be misbehaving, say dropping packets or changing Hop count or sequence number, umpire nodes U i , U i + 1 , and N i − 1 in the forward path and N i + 1 in the reverse path send a M-ERROR message to the source and set the status bit of guilty node N i to "1" indicating a red flag by M-Flag message. In these above two papers, we do not concentrate on selfish node. In this paper, token-based umpiring technique (TBUT) is proposed to detect and eliminate the selfish nodes efficiently in MANET. The main reason for using tokens in this analysis is to accelerate the detection and elimination of misbehaving selfish nodes. In MANET, nodes need to help other nodes to forward the data packets, but selfish nodes failed to do it. Because of selfishness of some nodes, the network performance may be reduced drastically.. A selfish node is a node that utilizes its limited resources, such as battery power, CPU time, and memory space purely for its own purpose. Because of its energy and storage constraints, all incoming data forwarding and route discovery packets are intensely not accepted by it, thereby it tries to save its own resources. The features of the selfish nodes are not forwarding the routing packets, not replying to the hello messages, postponing the route discovery packets, and not forwarding the data packets. The main objective of the proposed work is too detected and eliminates the selfish node in MANET using TBUT. The proposed method consists of a packet dropping detection mechanism and a selfish node quarantining mechanism. In packet dropping detection mechanism, the selfish node is traced and identified. A selfish node quarantine mechanism envisages marking the offending nodes so that they do not participate any further in the network activities. In this paper, we will explain our proposed TBUT. The rest of the paper is organized as follows: Section 2 provides models and assumptions, Section 3 gives an overview of the proposed model, Section 4 gives simulations and experimental results, in Section 5 we explore the related work, and Section 6 draws up conclusions. The security considerations in a TBUT setting are still in their infancy phase and require a more thorough analysis by the research community. The misbehavior of selfish nodes is a major problem in wireless MANET. The selfish nodes do not participate in the routing and data transmission process, which intentionally drop the packets. These misbehaviors of the selfish nodes will impact availability, efficiency, reliability, and fairness. The selfish node utilizes the resources for its own purpose, and it neglects to share the resources to other nodes. So, it is important to detect the selfish nodes in MANET. We have conducted simulation studies to evaluate the performance of TBUT in the presence of 30% selfish nodes and have compared it with ETUS routing protocols. The results show that TBUT significantly improves the performance of ETUS in all metrics, packet delivery ratio, and control overhead. The security analysis and experimental results have shown that TBUT is feasible for enhancing the security and network performance of real applications. In the following, a number of potential research directions are introduced. The key distribution center (KDC) architecture is the main stream in wired network because KDC has so many merits: efficient key management, including key generation, storage, and distribution and updating. The lack of a trusted third party (TTP) key management scheme is a big problem in ad hoc network [1, 5] . Different types of attacks on MANET were discussed by Abhay Kumar Rai et al. [19] ; they have designed a security mechanism by which they can minimize or completely remove many of those attacks. Sudha Rani et al. [20] proposed a detection and prevention of wormhole attack in stateless multicasting. Their scheme has no central administrator. They have shown that their schemes can handle wormhole attacks.. Leonidas Georgiadia et al. [21] made a survey of threats and possible solutions for resource allocation and crosslayer control in wireless networks. Raj et al. [22] proposed a solution for black hole attacks. It was implemented in prominent AODV protocol-based MANET. Tsou [23] developed a novel scheme BDSR to avoid black hole attack based on proactive and reactive architecture. Yu et al. [2] proposed a solution of a distributed and cooperative black hole node detection and elimination mechanism. Solda et al. [24] gave a solution for blacklisting attacks; in these papers, they studied the problem of forecasting attack sources based on past attack logs from several contributors. They formulated this problem as an implicit recommendation system [25, 26] .. Hernandez et al. [27] introduced a fast model to evaluate the selfish node detection in MANET using a watchdog approach. They estimated the time of detection and the overhead of collaborative watchdog approach for detecting one selfish node. Singh et al. [28] implemented a security-based algorithmic approach in MANETs. In this analysis, an empirical and effective approach was proposed to optimize the packet loss frequency. Jyoshna et al. [29] proposed a solution for byzantine attacks in ad hoc networks using SMT protocol that provides a way to secure message transmission by dispersing the message among several paths with minimal redundancy. Megha Arya and Yogendra Kumar Jain [30] gave a solution for gray hole attack. They use an intrusion detection system (IDS) to monitor the network or system, for selfish activities or policy violation, and produce reports to a management station. It takes over the sending of packets. Afterwards, the node just drops the packets to launch a (DoS) denial of service attack. If neighbor nodes that try to send packets over attacking nodes lose the connection to destination, then they may want to discover a route again by broadcasting RREQ messages [31] [32] [33] [34] .. B.B. Jayasingh and B. Swathi [35] proposed a mechanism that detects the jellyfish attacks at a single node and that can be effectively deployed at all other nodes in the ad hoc network. They gave a solution that detects the jellyfish reorder attack based on the reorder density which is a basis for developing a metric. Timothy et al.'s [36] paper focuses on jamming at the transport/network layer. Jamming at this layer exploits AODV and TCP protocols and is shown to be very effective in simulated and real networks when it can sense victim packet types, but the encryption is assumed to mask the entire header and contents of the packet so that only packet size, timing, and sequence is available to the attacker for sensing [19] .. Kurkure and Chaudhari [37] illustrated a comparative analysis of the selfish node detection methods based on detection time and message overhead. In this paper, a collaborative watchdog method was used to identify the selfish nodes and diminish the detection time and message overhead. Sahu and Sinha [38] suggested a cooperative approach for understanding the behavior of IDS in MANETs. In this paper, they described about various attacks and techniques used for intrusion detection which were proposed to provide high performance. Patel et al. [39] used an AODV protocol for trust-based routing in ad hoc networks. Ad hoc networks have limited physical security, less infrastructure, restricted power supply, mobility network, and changing network topology [40] [41] [42] [43] [44] . Jawhar et al. suggested a reliable routing protocol for enhanced reliability and security of communication in the MANET and sensor networks [45] .. Various P2P media streaming systems have been deployed successfully, and corresponding theoretical investigations have been performed on such systems [46] . In this paper, [47] thoroughly investigates the evolutionary dynamics of soft security mechanism, namely reciprocitybased incentive mechanism, in P2P systems based on evolutionary game theory (EGT). By soft security mechanism, it means social control mechanisms to overcome peers' selfish (rational) behaviors and encourage cooperation in P2P systems.. Trust management plays an important role in IoT [48] [49] [50] [51] [52] [53] [54] for reliable data fusion and mining, qualified services with context awareness, and enhanced user privacy and information security [9] . It helps people overcome perceptions of uncertainty and risk and engages in user acceptance and consumption on IoT services and applications [9, 55, 56] . However, current literature still lacks a comprehensive study on trust management in IoT [9] . Authenticated key agreement protocol is a useful cryptographic primitive, which can be used to protect the confidentiality, integrity, and authenticity for transmitted data over insecure networks [6] .. Built upon opportunistic routing and random linear network coding, CodePipe not only simplifies transmission coordination between nodes but also improves the multicast throughput significantly by exploiting both intra-batch and inter-batch coding opportunities [10] . In particular, four key techniques, namely LP-based opportunistic routing structure, opportunistic feeding, fast batch moving, and inter-batch coding, are proposed to offer substantial improvement in throughput, energy efficiency, and fairness [10] .. In the paper, [11] proposes a multi-constrained QoS multicast routing [12] method using the genetic algorithm. The proposal will be flooding limited [13] using the available resources and minimum computation time in a dynamic environment. By selecting the appropriate values for parameters such as crossover, mutation, and population size, the genetic algorithm improves and tries to optimize the routes.. For the author of this paper [57] , they consider the assignment strategy with topology preservation by organizing the mesh nodes with available channels and aim at minimizing the co-channel interference in the network. The channel assignment with the topology preservation is proved to be NP-hard and to find the optimized solution in polynomial time is impossible. They have formulated a channel assignment algorithm named as DPSO-CA which is based on the discrete particle swarm optimization and can be used to find the approximate optimized solution [57, 58] .. All the above schemes only try to protect the system from the attacker, but not bother about quarantining attackers [3, 16, 59] . The TBUT systems not only detect the mischievous nodes but also prevent their further participation in the network.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Progressive skylining over Web-accessible databases Skyline queries return a set of interesting data points that are not dominated on all dimensions by any other point. Most of the existing algorithms focus on skyline computation in centralized databases, and some of them can progressively return skyline points upon identification rather than all in a batch. Processing skyline queries over the Web is a more challenging task because in many Web applications, the target attributes are stored at different sites and can only be accessed through restricted external interfaces. In this paper, we develop PDS (progressive distributed skylining), a progressive algorithm that evaluates skyline queries efficiently in this setting. The algorithm is also able to estimate the percentage of skyline objects already retrieved, which is useful for users to monitor the progress of long running skyline queries. Our performance study shows that PDS is efficient and robust to different data distributions and achieves its progressive goal with a minimal overhead. Many database users are interested in the ''best'' queries-queries that return the objects that best fit a certain set of criteria. However, in many cases-especially in multi-criteria decision making-there is no single best answer to a query. Instead, we have a set of objects that satisfy some basic conditions but none of them is absolutely better than the others.. For example, suppose we want to book a hotel. We may specify the room price (the lower the better) and the distance of the hotel to the beach (the shorter the better) as the evaluation criteria. A hotel is interesting only if there does not exist another available hotel that is not more expensive, but closer to the beach, or one that is not further away from the beach, but costs less. Fig. 1 shows ten hotel options, where the two axes represent the two evaluation criteria in arbitrary scales. Following the arguments above, only options b, c, f and i (marked by solid circles) are interesting. We cannot claim any of them to be the best hotel without additional information about the relative weights of the two criteria. However, the best hotel is always in this set for all possible non-negative weights. We call such a set of objects the skyline (since the line connecting them resembles a skyline), the objects in the set the skyline objects, and the queries that retrieve all skyline objects the skyline queries [1] .. Skylines are useful for multi-criteria decision making as they represent sets of objects (e.g. solutions to a problem) out of which no better objects can be found. They can be used to filter suboptimal objects, after which users can focus on the small skyline and select the objects that fit their needs.. As a tremendous amount of data is spread all over the Internet, it is possible that each criterion (represented by an attribute of an object) is stored at a separate site. For our hotel example, the room prices and distances to the beach could be stored at a broker Web site and a digital map Web site respectively. It is thus important to devise algorithms that can answer this kind of distributed skyline queries. In real situations, it is very common for users to refine their queries several times before getting the desired results. For example, if a user observes the query results that based on the above two criteria contain some hotels that are too far away from the airport, he may refine the query by adding the time to the airport as a new evaluation criterion. A user may get frustrated if he realizes that his query needs to be refined after waiting for a long processing time. This means having a good response time is a crucial requirement for algorithms that answer skyline queries. Rather than returning all skyline objects at the end of the query processing, the algorithms should return skyline objects as long as they are ready since they may give important refinement hints to users, who may decide to terminate the current query and start a refined one immediately. We will devise one such progressive algorithm in this paper. We will also minimize the number of remote data access, as the cost of remote data access can be much higher than the computational cost [2, 3] . The algorithm can be useful in applications such as interactive search engines.. Pioneering algorithms have been proposed to find skylines in centralized databases in a batch [1] , and in a progressive fashion [4] [5] [6] . Ref. [2] extends skyline queries to the Web environment described above. However, as far as we know, there have been no previous works on finding skylines progressively in the distributed environment described above. In this paper we will propose an efficient progressive algorithm for processing skyline queries in such an environment.. The main contributions of this study are: (1) We present a novel algorithm, PDS (progressive distributed skylining), for evaluating skyline queries over Web data sources. Internally it uses R*-trees to efficiently determine whether an object belongs to the skyline. (2) We propose a new linear-regression based method to enable faster identification of skyline. (3) We extend PDS to evaluate top-K skyline queries over Web databases. (4) We propose ways to estimate the percentage of skyline objects already retrieved, which is useful for users to monitor the progress of long running skyline queries. (5) We show, by various experimental results, that PDS outperforms and is more robust than the distributed skyline algorithms in [2] in terms of both the number of source access and processing time.. The rest of the paper is organized as follows. Section 2 presents the related work on skyline query processing. In Section 3, we review the basic blocking skyline algorithms for the Web, which solves a problem similar to ours, but does not return skyline objects progressively. Section 4 describes the details of PDS. In Section 5, we discuss how to extend PDS to evaluate other types of skyline queries and to estimate the query progress. In Section 6, we report the experimental results and finally in Section 7, we conclude the study with some directions for future works. Skyline queries are important for database applications like decision support and customer information systems. In this paper, we proposed PDS, a novel progressive algorithm that computes skyline queries over Web-accessible databases. Users execute skyline queries over Web data sources by PDS and retrieve the results incrementally. As a result, users could make their decisions in real-time. By making use of a simple estimation based on linear regression and a standard index structure R*-tree, PDS evaluates skyline efficiently in terms of both the number of source access and the computational time over various kinds of data distributions. Furthermore, it does not require any pre-computation (the R*-tree is created on-the-fly) of the remote data and can also be easily extended to evaluate top-K skyline under the Web environment. In addition, PDS is user-friendly by supporting a progress estimation such that users know the headway of the query evaluation process and can make early decisions accordingly.. There are several avenues for future work. First, we plan to improve the usability of PDS by allowing users to barter between progressiveness and efficiency. It is because we found that the number of source access could be saved by reporting skyline points less frequently.. Second, distributed query processing on mobile devices has also drawn much attention in recent years [20] . Thus, we plan to evaluate distributed skyline on devices with tight memory requirements (e.g., PDA and cell phone). It would be useful and convenient if users could execute skyline queries through such devices. Skylines, and other directly related problems like maximum vector [7] , contour problem [8] and multi-objective optimization [9] , have been well studied for conventional datasets. However, the algorithms do not work when the dataset does not fit into main memory. In relational database context, Borzsonyi et al. [1] first proposed two algorithms, namely Divide-and-Conquer (D&C) and Block Nested Loops (BNL), to evaluate skyline queries. D&C divides the dataset into several partitions such that each partition can fit into memory. Skyline objects for each individual partition are then computed by a main-memory skyline algorithm. The final skyline is obtained by merging the skyline objects for each partition. BNL compares each object in a nested-loop by keeping a list of candidate skyline objects in main memory. Dominated objects are discarded during the comparisons. The efficiency of BNL is further improved in [10] by sorting the entire dataset according to a monotonic function a priori.. Tan et al. [6] presented two progressive skyline algorithms that require pre-processing. The first algorithm encodes the dataset as a collection of bitmaps. Decision on whether an object belongs to the skyline is based on some bit operations. The second algorithm organizes a set of d-dimensional objects into d lists such that an object o is assigned to list i if and only if its value at attribute i is the best among all attributes of o. Each list is indexed by a B-tree, and the skyline is computed by scanning the B-tree until an object that dominates the remaining entries in the B-trees is found.. Kossmann et al. [4] observed that the skyline problem is closely related to the nearest neighbor (NN) search problem. They proposed an algorithm that returns skyline objects progressively by applying nearest neighbor search on an R-tree indexed dataset. Papadias et al. [5, 11] surveyed most of the secondary-memory algorithms for centralized database and proposed an I/O optimal algorithm that returns skyline progressively. More recently, Chan et al. [12] studied the computation of skyline with partially-ordered domains. They proposed to transform each partiallyordered attribute into a pair of integer attributes (interval) so as to enable the use of index-based techniques as well as avoid the ''dimensionality curse'' problem [1] .. The above methods are designed for centralized databases and do not work well in the distributed Web environment because of two reasons. First, they assume attribute values can be accessed at no (or negligible) cost. It is true in a centralized database. However, running these algorithms in the distributed setting may incur substantial amount of unnecessary network access. Second, these centralized query processing techniques rely on indexes that build on all attributes involved in the queries. However, these centralized data structures are not available in a fully decentralized and distributed environment like the Internet.. Very recently, Lin et al. [13] studied ways to support on-line skyline query computation in a stream environment. Still, their work is different from what we are studying because the data values of an object arrived from a data stream are all known a priori, where it is not necessary true in the Web.. Following the general query model [3] for the Web environment, Balke et al. [2] extended the skyline problem for the Web in which the attributes of an object are distributed in different Webaccessible servers, as we have described in the last section. They proposed two algorithms that compute the skyline in such a distributed environment, which return all skyline objects at the end rather than progressively. Since the problem that they study is similar to the one of the current study, we will discuss them in detail in the next section as an important reference to our new algorithm.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O