Crossing-Preserving Coherence-Enhancing Diffusion on Invertible Orientation Scores Many image processing problems require the enhancement of crossing elongated structures. These problems cannot easily be solved by commonly used coherenceenhancing diffusion methods. Therefore, we propose a method for coherence-enhancing diffusion on the invertible orientation score of a 2D image. In an orientation score, the local orientation is represented by an additional third dimension, ensuring that crossing elongated structures are separated from each other. We consider orientation scores as functions on the Euclidean motion group, and use the group structure to apply left-invariant diffusion equations on orientation scores. We describe how we can calculate regularized left-invariant derivatives, and use the Hessian to estimate three descriptive local features: curvature, deviation from horizontality, and orientation confidence. These local features are used to adapt a nonlinear coherence-enhancing, crossing-preserving, diffusion equation on the orientation score. We propose two explicit finite-difference schemes to apply the nonlinear diffusion in the orientation score and provide a stability analysis. Experiments on both artificial and medical images show that preservation of crossings is the main advantage compared to standard coherenceenhancing diffusion. The use of curvature leads to improved enhancement of curves with high curvature. Furthermore, Image-processing problems often demand enhancement of elongated structures, such as lines, contours, and oriented textures, in noisy images. Many methods for enhancing elongated structures are based on nonlinear anisotropic diffusion equations on the image, i.e. adaptive diffusion processes where an unequal amount of diffusion is applied in different directions. This idea was pioneered by Nitzberg and Shiota (1992) and Cottet and Germain (1993) . Later on, Weickert proposed edge-and coherence-enhancing diffusion filtering (Weickert 1998 (Weickert , 1999 , which uses the structure tensor to steer the diffusion. Afterwards, various publications appeared inspired by these methods. For example, Manniesing and Niessen (2005) , Manniesing et al. (2006) proposed to steer the diffusion using the vessel resemblance function, which is based on the Hessian instead of the structure tensor, and Tschumperlé (2006) proposed to include curvature into the diffusion process in order to improve enhancement of curved structures.. Many image processing problems, especially medical ones, require the algorithm to handle crossing and bifurcating line structures appropriately. This is for example important for High Angular Resolution Diffusion Imaging (HARDI) (Tuch et al. 1999) , which is an extension to Diffusion Tensor Imaging (DTI), where a richer representation Fig. 1 Example of a microscopy image with crossing collagen fibers, showing the problems occurring at crossing. With the standard coherence-enhancing diffusion method, cf. Weickert (1998) , abbreviated as CED, crossing elongated structures are not preserved. The method described in this paper is able to handle crossings appropriately. The parameters that are used to obtain these images are given in Sect. 8 for angular data is used, such that crossing fibers can be distinguished. Other examples include microscopy images of for instance collagen structures, X-ray fluoroscopy images with catheters (Franken et al. 2006) , and MRI images of bifurcating blood vessels. At a position with a crossing, methods like coherence-enhancing diffusion do not enhance the elongated structures appropriately, as illustrated in Fig. 1(b) . One would resolve this problem if one diffuses anisotropically in the directions of the different oriented structures, independent on the angles and number of elongated structures that cross. An interesting approach to handle crossing elongated structures is proposed by Scharr (2006) . The method is similar to coherence-enhancing diffusion, but the gradient operator in both the structure tensor and the diffusion PDE is substituted by the second order jet operator. However, the drawback of this method is that the order of the PDE is high, and gets even higher if one wants to deal with crossings of more than two curves. Furthermore, it is only suitable to handle "X-junctions", i.e. 2 curves that cross with a large angle.. In this paper we try to resolve the problem of crossing elongated structures by using orientation scores of images. An orientation score of a two-dimensional image is a threedimensional function on spatial position (x, y) and orientation θ , i.e. a function on the 2D Euclidean motion group, which provides an overview of all local orientations in the corresponding image. This is illustrated in Fig. 2 . The concept of orientation scores first occurred in the field of perceptual grouping (Walters 1987; Heitger and von der Heydt 1993; Mumford 1994; Williams and Jacobs 1997; Zweck and Williams 2004) and is also applied for segmenting crossing structures (Chen et al. 2000; August 2001; Van Ginkel 2002) and estimating local orientation (Felsberg et al. 2006) . The main advantage of using an orientation score is that crossing curves are separated (see Figs. 2(a) and (b)), since at the position where two lines cross they have a different orientation. This means that it is not needed anymore to take special care for crossings e.g. by explicitly detecting them. Figure 1 (c) shows a typical result that can be obtained using orientation scores. Kalitzin et al. (1997) proposed the invertible orientation score, which makes it possible to reconstruct the original image from the orientation score in a well-posed way. The invertibility of the transformation is essential in order to use orientation scores (OS) for image processing and enhancement, using the following chain of operations Image to OS −→ Process OS −→ OS to image.. An example of a simple orientation score processing operation is to take a certain power or taking nonlinear combinations of derivatives in the orientation score, leading to enhancement of lines in the reconstructed image (Kalitzin et al. 1997) . Duits et al. (2004 , Duits (2005) developed a theory on the robustness of the invertible orientation score transformation, which is a multi-orientation wavelet transform. In wavelet literature, several authors have investigated wavelet transforms that are both multi-orientation and multiscale. The curvelet transform Donoho 1999a, 1999b; Starck et al. 2002 ) is a discrete multi-scale and multiorientation wavelet transform. Antoine et al. propose continuous directional wavelets (Antoine and Murenzi 1996; Antoine et al. 1999) and establish the wavelet theory on the similitude group. The orientation score wavelet transform leaves out scaling, while it is still possible to obtain a stable image reconstruction. 1 In fact, an orientation score wavelet picks up all scales simultaneously, i.e. all scales within a reasonable range determined by the sampling and the size of the image. A practical advantage is the reduction of storage requirements. 2 The notion of scale does come into play in orientation scores when considering processing operations on orientation scores. Instead of applying soft thresholds on the wavelet domain, which is common practice in wavelet literature, Duits (2005) explicitly employs the group structure in the wavelet domain by considering left-invariant convection-diffusion equations on the orientation score, which can be regarded as scale spaces on the Euclidean motion group. The linear left-invariant convection-diffusion equation correspond to stochastic processes for enhancement and completion of curves (Duits 2005 ; Van Almsick 1 The admissibility constraint in the wavelet theory on the similitude group (Antoine et al. 1999) can not be satisfied by proper wavelets in the Euclidean motion group. Therefore, we need a totally different admissibility constraint to construct proper wavelets in case of the Euclidean motion group (Duits et al. 2004 ), which will be briefly addressed in this paper. 2 Another reason to leave out scaling is that the admissibility conditions for multi-scale multi-orientation wavelet transforms (Antoine et al. 1999 ) require the oriented wavelet to oscillate along its radial direction, which is an undesirable effect that will cause problems for the diffusion processes in the wavelet domain that we will aim at in this paper.. 2007). The major differences with the earlier work in this area, i.e. the approach for curve enhancement by Citti and Sarti (2006) and curve completion by Mumford (1994) and Zweck and Williams (2004) , is that the invertibility of the orientation score is used. Furthermore, the linear evolution equations are solved by means of an SE(2)-convolution on the orientation score with the corresponding Green's function, which have been derived by Duits and van Almsick (2008) , Duits and Franken (2007, 2009a) . This paper goes one step further and describes nonlinear diffusion equations on the two-dimensional Euclidean motion group to enhance crossing elongated structures. The method is comparable to edge-and coherence-enhancing diffusion, with the difference that the orientation is explicitly encoded in the domain on which diffusion is applied. We describe how to make the diffusion tensor adaptive to three features that describe the local structure in the orientation score: orientation confidence, curvature, and deviation from horizontality. These features are estimated using a Hessian matrix and they establish a gauge frame for the diffusion process at each position in the domain of the orientation score. The idea of a gauge frame in image processing (Florack et al. 1993 ) is to define, at each spatial position independently, a data-dependent orthogonal coordinate frame that spans the tangent space, such that the basis vectors are in alignment with some local feature of interest in the image. This paper extends earlier work presented at conferences (Franken et al. 2007a (Franken et al. , 2007b with new results, particularly the gauge frame, stability bounds of the numerical schemes, and the concept of deviation from horizontality. In this paper we focus on the applied image processing and algorithmic part of our work. The essential mathematical theory will be presented in such a way that it should be understandable without too many prerequisites. In the companion papers (Duits and Franken 2009a, 2009b) , which are written by the same authors, we focus on the mathematical theory and underlying differential geometry of both linear and nonlinear left-invariant diffusion equations on orientation scores. In this paper we introduced nonlinear diffusion on invertible orientation scores. Starting from a 2D image, we constructed a three-dimensional orientation score using rotated versions of a directional quadrature filter. Since an orientation score is a function on the Euclidean motion group SE(2), we considered left-invariant diffusions adhering to the structure of this group. Then, we introduced a gauge coordinate frame that is used to formulate an anisotropic diffusion process that is aligned with exponential curves in SE(2).. We showed how one can use normal Gaussian derivatives to calculate regularized derivatives in the orientation score.. These Gaussian derivatives are used to estimate a tangent vector, which is tangent to the locally best fitting exponential curve, on each position in the orientation score. Using this tangent vector we calculate three features describing the local structure in the orientation score: curvature κ, deviation from horizontality d H , and orientation confidence s. The features κ and d H uniquely determine the optimal gauge frame. The nonlinear diffusion is aligned with this optimal gauge frame and the orientation confidence s controls whether locally the diffusion is isotropic or anisotropic.. We proposed two explicit numerical schemes to apply the nonlinear diffusion on the orientation score, and derived sharp stability bounds for both of these schemes. The simple explicit finite difference scheme is efficient but is not optimal concerning rotational invariance and oscillations at the Nyquist frequency. The left-invariant explicit finite difference scheme with spline interpolation, on the other hand, improves the left-invariance, but becomes very inefficient if deviation from horizontality is included.. The experimental results show that the curvature estimates are reliable. Furthermore we showed that we are indeed able to enhance elongated structures in images and that including curvature helps to enhance lines with large curvature. Especially at crossings our method renders a more natural result than coherence enhancing diffusion. The adaptation of the diffusion to deviation from horizontality d H helps to get sharper results without orientational biases towards the sampled orientations, especially if we lower the number of orientations N o . Therefore, including deviation from horizontality enables a large reduction in computations and storage. The diffusion shows the typical nonlinear scale-space behavior when increasing time: blurring occurs, but the important features of images are preserved over a longer range of time.. Some issues could be addressed in future work. The numerical schemes that are proposed should be improved concerning computational speed and quality. Furthermore, it would be interesting to use the same approach to other groups (Duits and Burgeth 2007) such as the Heisenberg group, for the purpose of image enhancement via the Gabor domain, or the similitude group to use multi-scale and multiorientation simultaneously. Finally, it is interesting to apply the same techniques in the 3D Euclidean motion group, to enhance elongated structures in three-dimensional images. This is especially useful for enhancing and segmenting fibers in High Angular Resolution Diffusion Imaging (HARDI) data. This is currently being investigated by the authors of this paper (Franken 2008, Chap. 7) . This section is divided in two parts. First we will quantitatively evaluate the quality of the curvature estimation, using a set of artificial test images. Then we will show results of the coherence-enhancing diffusion in orientation score (CED-OS) algorithm. We will qualitatively show the differences between including curvature and/or deviation from horizontality. We will also compare CED-OS with standard coherence-enhancing diffusion on the image.

O B I O O B I O O O O O O O O B I I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O B I O O O O B I I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I I O O B I I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O


New Results on Noncommutative and Commutative Polynomial Identity Testing Abstract. Using ideas from automata theory, we design the first polynomial deterministic identity testing algorithm for the sparse noncommutative polynomial identity testing problem. Given a noncommuting black-box polynomial f ∈ F{x 1 , . . . , x n } of degree d with at most t monomials, where the variables x i are noncommuting, we give a deterministic polynomial identity test that checks if C ≡ 0 and runs in time polynomial in d, n, |C|, and t. Our algorithm evaluates the black-box polynomial for x i assigned to matrices over F and, in fact, reconstructs the entire polynomial f in time polynomial in n, d and t. We apply this idea also to the reconstruction of black-box noncommuting algebraic branching programs (considered by Nisan (1995) and Raz and Shpilka (2005) ) and obtain some results and connections to the problem of exact learning of noncommuting ABPs. Finally, we turn to commutative identity testing and explore the complexity of the problem when the coefficients of the input polynomial come from an arbitrary finite commutative ring with unity whose elements are uniformly encoded as strings and the ring operations are given by an oracle. We show that several algorithmic results for polynomial identity testing over fields also hold when the coefficients come from such finite rings. Polynomial identity testing (denoted PIT) over fields is a well studied algorithmic problem: given an arithmetic circuit C computing a polynomial in F[x 1 , x 2 , . . . , x n ] over a field F, the problem is to determine whether the polynomial computed by C is identically zero. The problem is also studied when the input polynomial f is given only via black-box access that allows the evaluation of the polynomial f at any point in F n or in F n for a field extension F of F. When f is given by a circuit, the problem is in randomized polynomial time (DeMillo & Lipton 1978; Schwartz 1980; Zippel 1979) , and these algorithms work even in the black-box setting, as long as |F| is suitably larger than deg(f ). It is a major challenge to obtain deterministic polynomial time algorithms even for restricted versions of the problem. The results of Kabanets & Impagliazzo (2004) show that the problem is as hard as proving superpolynomial circuit lower bounds. Indeed, the problem remains open even for depth-3 arithmetic circuits with an unbounded + gate as output (Dvir & Shpilka 2005; Kayal & Saxena 2007) .. As shown by Nisan (1991) , it is somewhat easier to prove lower bounds for noncommutative algebraic computation. Using a rank argument, Nisan has shown exponential size lower bounds for noncommutative formulas (and noncommutative algebraic branching programs) that compute the noncommutative permanent or determinant polynomials in the ring F{x 1 , . . . , x n } where x i are noncommuting variables. Thus, it seems plausible that identity testing in the noncommutative setting ought to be easier too.. Indeed, Raz & Shpilka (2005) have shown that for noncommutative formulas (and algebraic branching programs) there is a deterministic polynomialtime algorithm for polynomial identity testing. However, for noncommutative circuits the situation is somewhat different. Bogdanov & Wee (2005) show using the Amitsur-Levitzki theorem that identity testing for polynomial degree noncommutative circuits is in randomized polynomial time. Basically, the Amitsur-Levitzki theorem allows them to randomly assign elements from a matrix algebra M k (F) for the noncommuting variables x i , where 2k exceeds the degree of the circuit. Here, it is interesting to note that, when the degree of the circuit is not restricted to polynomial, the Bogdanov-Wee algorithm yields only a randomized exponential time upper bound. It is open whether this upper bound can be improved.. The main contribution of this paper is the use of ideas from automata theory to design an efficient deterministic polynomial identity tests for blackbox noncommutative polynomials that are sparse (i.e, have only polynomially many monomials) and of polynomial degree. Indeed, our algorithm can even reconstruct such a polynomial. More precisely, suppose we are given a noncommutative black-box polynomial f (x 1 , x 2 , . . . , x n ) of degree d with t monomials in F{x 1 , . . . , x n }, where the variables x i are noncommuting. This means that we can assign matrices over F to the variables x i and evaluate the black-box polynomial f , where the black-box will give the resulting matrix value of the polynomial f . Then we give a deterministic polynomial-time algorithm that can reconstruct the entire polynomial f in time polynomial in d, n, and t.. We note that the sparse polynomial identity testing and polynomial interpolation problems in the setting of commuting variables (i.e. for polynomials in the commutative ring F[x 1 , x 2 , . . . , x n ]) have been considered and solved in a long line of works (Ben-Or & Tiwari 1988; Clausen et al. 1991; Grigoriev & Karpinski 1987; Grigoriev et al. 1990; Klivans & Spielman 2001) . However, it is not clear how to adapt these solutions to the noncommutative case.. The key idea in our algorithm is to consider the noncommuting monomials over the x i as words and to design finite automata that allow us to distinguish between different words. Then, using the connection between automata, monoids and matrix rings we are able to deterministically choose a relatively small number of matrix assignments for the noncommuting variables to decide if C ≡ 0. Thus, we are able to avoid using the Amitsur-Levitzki theorem used in Bogdanov & Wee (2005) . Indeed, we can easily derive an alternative automata-theoretic proof of a (weaker) version of the Amitsur-Levitzki theorem which gives another correctness proof for the randomized polynomial-time identity test in Bogdanov & Wee (2005) for noncommutative circuits.. We also apply this idea to black-box noncommutative algebraic branching programs. We observe interesting connections between identity testing of black-box noncommutative ABPs, reconstruction of black-box noncommutative ABPs, and the problem of exact learning automata. As a first step towards obtaining deterministic algorithms, we also give a polynomial-time deterministic reconstruction algorithm for black-box noncommutative ABPs where our black-box model assumes that we can query for the output of any specified gate of the ABP, not just the output gate.. As stated above, we obtain polynomial-time algorithms for sparse noncommutative polynomial identity testing for arithmetic circuits of polynomial degree. Note that sparseness is indeed a restriction on the polynomial computed by the input circuit -even with the polynomial degree assumption, the polynomial computed by a noncommutative arithmetic circuit may in general contain exponentially many monomials. The problem of deterministically identity 524 Arvind, Mukhopadhyay & Srinivasan cc 19 (2010) testing general noncommutative arithmetic circuits, even under the restriction of polynomial degree, remains open. But can we hope for a deterministic polynomial-time PIT algorithm for noncommutative circuits of polynomial degree? For commutative circuits, Kabanets & Impagliazzo (2004) have shown that derandomizing PIT implies circuit lower bounds: an efficient deterministic algorithm for PIT would tell us that either NEXP ⊆ P/poly or the integer Permanent does not have polynomial-size arithmetic circuits. We observe in Section 4 that a similar result also holds in the noncommutative setting. More precisely, if noncommutative PIT has a deterministic polynomial-time algorithm then either NEXP ⊆ P/poly or the noncommutative Permanent polynomial does not have polynomial-size noncommutative circuits.. In the last section of the paper, we study commutative polynomial identity testing in the more general setting when the coefficients of the polynomials come from a commutative ring R with unity instead of a field. The question here is how precisely the complexity of identity testing depends on the structure of the ring R. Suppose we restrict ourselves to finite commutative rings R, where we assume that the elements of R are uniformly encoded as strings in {0, 1} m with two special strings encoding 0 and 1, and the ring operations are carried out by queries to the ring oracle. Then it turns out that the internal algebraic structure of R does not change the complexity of PIT. It suffices that the elements of R have polynomial-size encoding, and w.r.t. this encoding the ring operations can be efficiently carried out. In this general setting we show a suitable generalization of the Schwartz-Zippel test which yields a randomized polynomial-time identity test.. 1.1. Organization. The paper is organized as follows. In Section 2, we study the noncommutative black-box identity testing for sparse and small degree polynomials. We also show an interpolation algorithm for such polynomials. In Section 3, we consider the problems of identity testing and interpolation of black-box noncommutative ABPs and discuss connections to the exact learning model. We describe an interpolation algorithm for ABPs in a stronger model. In Section 4, we discuss the consequence of the derandomization of identity testing in the noncommutative model and observe results analogous to those of Kabanets & Impagliazzo (2004) . From Section 5 onwards, we discuss commutative and noncommutative identity testing over finite rings. In Section 5, we prove an analogue of the Schwartz-Zippel lemma for finite commutative rings with unity. The identity testing results are given in the Section 6.

O O O O O O B I I O O O O B I O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Object-based and class-based Composition of Transitive Mixins In object-oriented composition, classes and class inheritance are applied to realize type relationships and reusable building blocks. Unfortunately, these two goals might be contradictory in many situations, leading to classes and inheritance hierarchies that are hard to reuse. Some approaches exist to remedy this problem, such as mixins, aspects, roles, and meta-objects. However, in all these approaches, situations where the mixins, aspects, roles, or meta-objects have complex interdependencies among each other are not well solved yet. In this paper, we propose transitive mixins as an extension of the mixin concept. This approach provides a simple and reusable solution to deWne "mixins of mixins". Moreover, because mixins can be easily realized on top of aspects, roles, and meta-objects, the same solution can also be applied to those other approaches. In many object-oriented approaches, the (multiple-)inheritance relationship and the type concept are modeled via the same construct, the class. However, (multiple-)inheritance primarily aims at the reusability of classes, whereas a class primarily deWnes the type of its instances, the objects (see also [8] ). These two goals are often contradictory, as, on the one hand, a unit of reuse should be small and Xexibly composable with arbitrary kinds of other classes, and, on the other hand, an object's type needs to be deWned completely and requires a Wxed place in the class hierarchy. Mixins are proposed as a way to solve this problem (see, e.g. [3, 5, 6, 26, 29, 41] ). A mixin is a small unit of composition that is not necessarily deWned completely. It can be mixed into a given class hierarchy at arbitrary places.. An open issue in mixin-based composition is the composition of multiple mixins in dependency to each other, i.e., how to deWne "mixins of mixins". As mixins can be used in arbitrary places of a class hierarchy, it is hard to deWne the interdependencies between them in the context of compositions in a generic way.. Consider a simple example: an access control handler is conditionally composed with application logic classes in a server. The access control handler depends on a number of other classes: for example the remote objects which have to be protected, the users and/or roles whose access requests to objects must be controlled, the permissions of each particular user or role, and the context constraints for these permissions. 1 As these classes are all together deWning the access control handler type, and each of them should be Xexibly composable and reusable in many situations, it seems to be a good choice to model each of them as a mixin. In a "Xat" mixin model, however, we are not able to model the interdependencies among these mixins. Example problems in such models are that conditional composition based on runtime state is not possible, the composition order cannot be speciWed, multiple roles of one type cannot have diVerent instance-speciWc permissions, or all mixins would be applied to the user or role instances. Instead, we would like to be able to explicitly model a kind of "mixin of mixin" relationship: a role mixin might only be conWgured for users, a permission mixin only for roles, and a context constraint mixin only for permissions. In such cases, the problem arises how these interdependencies among the classes can be properly modeled while still retaining the reusable type relationship oVered by the mixin concept.. In this paper, we propose the transitive composition of mixin classes as a solution to this problem. In particular, in our basic concept, which is called transitive mixin chains, each mixin can transitively have other mixins itself, to model (and arbitrarily reWne) mixin-based compositions. This way, multiple class hierarchies, expressing orthogonal concerns, can be (dynamically) composed with the application logic in a transitive fashion. Moreover, we also introduce the more elaborate transitive mixin delegation concept. It allows each transitive mixin to have its own (object-speciWc) state. These concepts are deWned in a generic way using Horn clauses in Section 3.. Our approach applies mixins as a simple basic concept for reusable types. We used this approach because of the generality of the mixin concept. Similar concepts are present in many recent adaptation techniques, including aspect-oriented programming, meta-object protocols, roles, message interceptors, interpreters, virtual machines, etc. That is, our approach can also be implemented as an extension to these other techniques, and hence we expect a wide applicability of our concepts.. Section 4 presents a proof-of-concept implementation of our approach that is based on XOTcl mixins [29, 30] . Subsequently, in Section 5, we illustrate the practical use of the concepts with two case studies, a persistence manager component and a role-based access control framework. We present these details, because we feel that -even though the concept in general and its use are quite simple and straightforward -the implementation details are not obvious. We illustrate the general problems in implementing transitive mixin classes by explaining the design challenges and decisions of our implementation, as well as the corresponding case studies. Of course, many design challenges can be solved quite diVerently in other implementations of our concepts. In Section 6 we evaluate our Wndings and Section 7 concludes the paper. In this paper, we have presented a practical approach to model mixin interdependencies. By applying mixin classes transitively, we are able to use the concept of mixin classes to deWne composition relationships of ordinary classes and mixins. Problems similar to the problem to deWne "mixins of mixins" are present in many other composition approaches as well -such as in aspect-oriented programming, meta-object protocols, roles, message interceptors etc. Hence, there is a broad applicability of the transitive mixin approach. The mixin concept is a rather simple extension to the basic type concepts of object-oriented languages, and is thus well suited to explore the problems of class relationships and interdependencies generally and conceptually -apart from the implementation details of the other composition approaches. We did two proof-of-concept implementations, XOTcl and Frag, which are both available as open source.. Moreover, our mixin concepts have been successfully applied in a number of projects (including the two case studies presented in this paper). As future work, we plan to implement the concepts as an extension of an existing AOP framework. In this paper, we focused on the extension of programming frameworks or languages by transitive mixin classes. As further work we also plan to provide modeling support for the concepts presented in this paper, for instance using a UML 2 extension.

O O O O O O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O B O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O


Speeding up search in peer-to-peer networks with a multi-way tree structure Peer-to-Peer systems have recently become a popular means to share resources. Effective search is a critical requirement in such systems, and a number of distributed search structures have been proposed in the literature. Most of these structures provide "log time search" capability, where the logarithm is taken base 2. That is, in a system with N nodes, the cost of the search is O(log2N ). In database systems, the importance of large fanout index structures has been well recognized. In P2P search too, the cost could be reduced considerably if this logarithm were taken to a larger base. In this paper, we propose a multiway tree search structure, which reduces the cost of search to O(logmN ), where m is the fanout. The penalty paid is a larger update cost, but we show how to keep this penalty to be no worse than linear in m. We experimentally explore this tradeoff between search and update cost as a function of m, and suggest how to find a good trade-off point. The multi-way tree structure we propose, BATON*, is derived from the BATON structure that has recently been suggested. In addition to multi-way fanout, BATON* also adds support for multi-attribute queries to BATON. Peer-to-peer (P2P) systems have become very popular of late, and are widely used for sharing resources, such as music files. Search is a crucial operation in P2P systems, and there has been considerable recent work in devising effective * Supported in part by the US National Science Foundation grant IIS-0219513 † Supported in part by IDA CCC grant as part of BestPeer [17] project distributed search techniques. The proposed structures include a ring as in Chord [13] , a multiple dimensional grid as in CAN [18] , a multiple list as in SkipGraph [4] , or a tree as in BATON [11] . Most search structures, including all the ones just mentioned, bound the cost of the search to a logarithm of the search space: for a system with N nodes, the search cost is bounded at O(logN). However, upon closer examination we see that this logarithm is to base 2. If we could take the logarithm to a larger base, we could have a substantially smaller cost.. The importance of having a large base for the logarithm has been well-recognized in centralized database systems. When tree structures are constructed for indexing, the fanout is made as large as possible, within the limits of the page size. We always use B-trees in database systems, and not binary trees! An equivalent transformation, from the low fanout structures currently proposed in the literature, to practical high fanout distributed search structures, is the topic of this paper.. At first glance, one may think that it is not too hard to increase the fanout in any distributed search structure. For instance, consider Chord. Rather than doubling successive ranges, we can increase them by a factor of m; correspondingly, in the finger table we would have m − 1 entries at each level, instead of just one entry. With this simple change, the search cost (in number of hops required) has gone down by a factor of log2m. The storage requirement, for the finger tables, has increased by a factor of m/log2m, but the amount of storage for finger tables is usually not large enough that this matters. So we appear to have won. The difficulty is that the update cost has gone up by a factor of (m/log2m). 2 . This quadratic dependence on m makes it difficult to increase m very much. We seek a better solution, preferably one in which the cost for updates increases no worse than linearly with m.. The only existing distributed index that is notable in permitting high fanout is the P-Tree [8] . In this proposal, each P2P node maintains a leaf node in the B + -tree and a path of virtual index nodes from the root to that leaf node. Search is very effective, but updates are expensive, possibly requiring substantial synchronization effort. In other words, this technique too has the same weak spot as multi-way Chord discussed above.. In this paper, we propose a new multi-way distributed tree structure. We call it BATON*, since it is an extension of BATON [11] , just as R*-trees are an extension of R-trees. BATON* extends BATON to permit a fanout of m > 2. With this, the cost of search becomes O(logmN ), as expected. Moreover, the cost of updating routing tables is just O(m · logmN ), as compared to O(log2N ) in BATONa degradation that is better than linear in m, and therefore a cost one may be willing to pay. Furthermore, BATON* has better fault tolerance properties than BATON, and is capable of quicker load balancing. In fact, the system's fault tolerance, measured as the number of nodes that must fail before the network is partitioned, increases linearly with m. Similarly, the expected cost of load balancing decreases linearly with m. We analyze the impact of m on both search and maintenance cost, and derive an estimation model for m that reduces the total cost given some composition of queries and updates. Likewise, we analyze the impact of m on fault tolerance of BATON*.. BATON* can also support queries over multiple attributes in an effective way. In addition to permitting the use of multiple attributes in a single index, BATON* further introduces the notion of attribute classification, based on importance of the attribute for querying, and the notion of attribute groups. Another variation of BATON [12] has been proposed for supporting multi-dimensional queries. However, as analyzed in Section 5, it cannot support multiattribute queries efficiently. Realizing that most of the time users only query over a small number of attributes while others are rarely queried, we propose a flexible method of indexing attributes in which only attributes, which are frequently queried, should be indexed separately; other attributes can be divided into small groups, and these groups are indexed as in multi-dimensional space.. To sum up, in this paper, we have the following major contributions:. • In Section 3, we introduce BATON*, a significant extension of BATON to multi-way tree structure. Therefore, we can reduce the cost for search from O(log2N ) to O(logmN ).. • We analyze BATON*'s resilience towards node failure and network partitioning and its capability of load balancing.. • Using BATON*, we present a flexible method to support multi-attribute queries efficiently in Section 5.. • In Section 6, we show the experimental results of our proposed system over PlanetLab [7] , a testbed for largescale distributed systems.. In addition to the main sections mentioned above, this paper includes necessary background information about BA-TON in Section 2; in Section 4, there is a description of how Chord can be extended to have a fanout of m; we present related work in Section 7 and conclude in Section 8. In this paper, we proposed a balanced multi-way tree structure, BATON*, which allows us to reduce the cost of routing message to O(logmN ). Additionally, by increasing the fanout of the tree, we increase the number of links among nodes, and hence increase the system's fault tolerance. Moreover, increasing the fanout leads to an increase in the number of leaf nodes, which facilitates better load balancing. Over this structure, we proposed a method to support multi-attribute queries efficiently. Our method relies on the construction of multiple independent indices for groups of one or more attributes. We suggest techniques for partitioning attributes into such groups. A careful experimental analysis, on the PlanetLab [7] infrastructure, confirms efficiency of our proposed system.. We have incorporated BATON* into BestPeer [17] and as a future work, we shall evaluate BATON* as part of BestPeer on data sharing applications. Extensive work has been done to support search in the area of distributed data. Most of them are based on scalable and distributed data structures (SDDS) such as [16] . However, these structures cannot be used in P2P systems where there is neither global index nor centralized servers. Since our paper focuses on search in P2P systems, we only discuss related work in P2P systems. Since our proposed multi-way structure is adapted from BATON (BAlanced Tree Overlay Network) [11] , we present here a brief review of the relevant features of BATON.. BATON is an overlay structure based on the binary balanced tree in which each peer in the network maintains a node of the tree. A node may connect to other nodes by up to four different kinds of links: parent links pointing to parent nodes, children links pointing to child nodes, adjacent links pointing to adjacent (in linear order) nodes that maintain adjacent ranges of values, and neighbor links pointing to selected neighbor nodes at the same level and have a distance equal to a power of two from the node. In BATON, each node in the tree, both leaf and internal, is assigned a range of values in which the range of values directly managed by a node is required to be greater than the range of values managed by its left adjacent node while smaller than the range of values managed by its right adjacent node. In other words, if we travel from the left to the right of the tree following adjacent links, data is in increasing order. An example of BATON is shown in Figure 1 . In BATON, a tree is considered balanced if and only if at any node in the tree, the height of its two subtrees differ by at most one. There are two important results in BATON. The first result shows that a tree is balanced if every node in the tree that has a child also has both its left and right routing tables full, i.e., none of the valid links in the routing table is Null. The second result shows that if a node x contains a link to another node y in its left or right routing tables, the parent node of x must also contain a link to the parent node of y unless the same node is parent of both x and y. This result gives an efficient way to forward requests among nodes in the network.. Based on these two key results, BATON keeps the tree structure balanced by forcing every node in the tree that has a child to have both its left and right routing tables full. Therefore, when a node receives a Join request from a new node, it can only accept the new node as its child if its routing tables are full and it does not have two children. Otherwise, the Join request is forwarded to either (a) its parent if at least one of its routing tables is not full or (b) one of its adjacent nodes if it has both routing tables and children full. In case of node departure, if the departure of a node does not affect the tree balance, the node can safely leave the network. This is the case where the departing node is a leaf node and all of its neighbor nodes have no children. Otherwise, the departing node has to find a replacement node, which is a node satisfying the previous case, to take its place. The replacement node is found by sending a FindReplacement request downward the tree. Here, the cost of finding a position for a new node and the cost of finding a replacement node in case of node departure are all O(logN) while the costs of updating routing tables to reflect changes are 6 · logN and 8 · logN for node join and node departure respectively.. In BATON, when a node receives a query request, if the searched value does not fall into its own range of values, the request is always forwarded to a node in its left routing table whose upper bound is still greater than the searched value or a node in its right routing table whose lower bound is still lower than the searched value if such a node exists. Otherwise, the query request is forwarded to either its left child/right child or its left adjacent/right adjacent node. Note that a query request can only be forwarded to a higher level node in two cases: (1) the higher level node contains the searched value, and hence it is necessary, (2) the processing node is a lower level leaf node without children and there are insufficient neighbor nodes inside its routing tables (in this case it is far from the root). As a result, BATON can avoid a bottleneck at the root and nodes near it.. Two load balancing schemes are used in BATON. In the first scheme, an overloaded/underloaded node performs load balancing with its adjacent nodes. In the second scheme, an overloaded/underloaded leaf node performs load balancing with a far away leaf node (a lightly loaded node may forcibly leave its current position and force-join as a child of the overloaded node to share the workload). The tree structure may become unbalanced in this case, and network restructuring becomes necessary. In BATON, network restructuring operations are akin to rotations in an AVL tree. Load balancing is performed by shifting nodes via adjacent links towards the place causing imbalance. BATON creates a binary search tree structure. As we attempt to increase the fanout, we find some challenges that must be overcome: Figure 2 shows a comparison between our proposal and this method in doing load balancing. The result shows that this naive method is very bad in solving load balancing problem.

O O O O B O O O B I I B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Digitally Continuous Multivalued Functions, Morphological Operations and Thinning Algorithms In a recent paper (Escribano et al. in Discrete Geometry for Computer Imagery 2008. Lecture Notes in Computer Science, vol. 4992, pp. 81-92, 2008 we have introduced a notion of continuity in digital spaces which extends the usual notion of digital continuity. Our approach, which uses multivalued functions, provides a better framework to define topological notions, like retractions, in a far more realistic way than by using just single-valued digitally continuous functions.In this work we develop properties of this family of continuous functions, now concentrating on morphological operations and thinning algorithms. We show that our notion of continuity provides a suitable framework for the basic operations in mathematical morphology: erosion, dilation, closing, and opening. On the other hand, concerning thinning algorithms, we give conditions under which the existence of a retraction F : X −→ X \ D guarantees that D is deletable. The converse is not true, in general, although it is in certain particular important cases which are at the basis of many thinning algorithms. The notion of continuous function is a fundamental concept in the study of topological spaces. For dealing with digital spaces, several approaches to define a reasonable notion of continuous function have been proposed. The first one goes back to A. Rosenfeld [19] in 1986. He defined continuous functions in a way similar to that used for continuous maps in R n . It turned out that continuous functions agreed with functions taking 4-adjacent points into 4-adjacent points. He proved, amongst other results, that a function between digital spaces is continuous if and only if it takes connected sets into connected sets. Independently of Rosenfeld, L. Chen [5, 6] seems to have developed the same notion of continuity, using the terms immersion, gradually varied operator, and gradually varied mapping. Chen's work appeared originally in Chinese.. More results related to this type of continuity were proved by L. Boxer in [1] and, more recently, in [2] [3] [4] . In these papers, he introduced such notions as homeomorphism, retracts, and homotopies for digitally continuous functions, applying these notions to define a digital fundamental group, digital homotopies, and to compute the fundamental group of sphere-like digital images. However, as he recognizes in [3] , there are some limitations with the homotopy equivalences he gets. For example, while all simple closed curves are homeomorphic and hence homotopically equivalent with respect to the Euclidean topology, in the digital case two simple closed curves can be homotopically equivalent only if they have the same cardinality.. A different approach was suggested by V. Kovalevsky in [17] , using multivalued functions. This seems reasonable, since an expansion such as f (x) = 2x must take 1 pixel to 2 pixels if the image of an interval has still to be connected. He calls a multivalued function continuous if the pre-image of an open set is open. He considers, however, that another important class of multivalued functions is what he calls "connectivity preserving mappings." By its proper definition, the image of a point by a connectivity preserving mapping is a connected set. This is not required for merely continuous functions. He finally asserts that the substitutes for continuous functions in finite spaces are the simple connectivity preserving maps, where a connectivity preserving map f is simple if for any x such that f (x) has more than 1 element then f −1 f (x) = {x}. However, in this case it would be possible to map the center of a 3 × 3 square to the k-boundary of it leaving the points of the k-boundary fixed, obtaining in this way a "continuous" retraction from the square to its k-boundary, something impossible in the continuous realm.. The multivalued approach to continuity in digital spaces has also been used by R. Tsaur and M. Smyth in [23] , where a notion of continuous multifunction for discrete spaces is introduced: A multifunction is continuous if and only if it is "strong" in the sense of taking neighbors into neighbors with respect to the Hausdorff metric. They use this approach to prove some results concerning the existence of fixed points for multifunctions. However, although this approach allows more flexibility in the digitization of continuous functions defined in continuous spaces, it is still a bit restrictive, as shown by the fact that the multivalued function used by them to illustrate the convenience of using multivalued functions is not a strong continuous multifunction.. In a recent paper [7] the authors presented a theory of continuity in digital spaces, using multivalued functions, which extends the one introduced by Rosenfeld and provides a framework to define topological notions, like retractions, in a far more realistic way than by using just single-valued digitally continuous functions. In particular, the deletion of simple points, one of the most important processing operations in digital topology, is characterized as a particular kind of retraction.. In this work we look more deeply into the properties of this family of continuous functions, now concentrating on morphological operations and thinning algorithms.. In Sect. 1 we revise the basic notions of digital topology required throughout the paper. In particular we recall different adjacency relations used to model digital spaces. In Sect. 2 we revise Rosenfeld's notion of digitally continuous function. In Sect. 3 we introduce the notion of subdivision of a topological space used to define continuity for multivalued functions and show some basic properties concerning the behavior of digitally continuous multivalued functions under restriction and composition. In Sect. 4 we show that the basic morphological operations of dilation and closing are continuous functions. We also show that, although the dual operations of erosion and opening cannot be modeled as continuous functions, they are so if we consider them defined on the set of white pixels. In Sect. 5 we show that the deletion of simple points can be completely characterized in terms of digitally continuous multivalued functions, and in Sect. 6 we extend this result to simple pairs. In the last sections we characterize thinning algorithms in terms of digitally continuous multivalued functions. Specifically, we show that the existence of an (N , k)-retraction F : X −→ X \ D guarantees that D is 4-deletable (respectively, 8-deletable) whenever D is made of 4-boundary (respectively, 8-simple) points. The converse is not true in general although it holds in certain particular important cases which are at the basis of many thinning algorithms.. For information on Digital Topology we recommend the survey [15] and the books by Kong and Rosenfeld [16] , and by Klette and Rosenfeld [13] . Other results on the discretization of topological notions can be found in [9, 10] . As an illustration of the usefulness of topological results, as those presented in this paper, in applications, we refer the reader to [11, 21] , where the combination of continuous and digital topological notions has allowed the development of algorithms to solve important problems, such as topological constrained segmentation. We have presented in this paper a theory of continuity in digital spaces, using multivalued functions, which extends the one introduced by Rosenfeld and provides a framework to define topological notions in a far more realistic way than by using just single-valued digitally continuous functions.. In particular, we have shown that the basic morphological operations of dilation and closing are continuous functions and that, although the dual operations of erosion and opening cannot be modeled as continuous functions on the set of black pixels, they are so if we consider them defined on the set of white pixels.. On the other hand, the deletion of simple points and simple pairs can be completely characterized in terms of digitally continuous multivalued functions. This fact has been used to characterize thinning algorithms in terms of digitally continuous multivalued functions. We have shown that the existence of an (N , k)-retraction F : X −→ X \ D guarantees that D is 4-deletable (respectively, 8-deletable) whenever D is made of 4-boundary (respectively, 8-simple) points. The converse is not true in general although it holds in certain particular important cases which are at the basis of many thinning algorithms.. The notion and main properties of a digitally continuous multivalued function are presented for any dimension. The results on retractions at the end of Sect. 3 are stated for Z 2 , since their main role is that of justification of the appropriateness of our notion of continuity, although they are easily extendable to higher dimension. The same is true for the results on mathematical morphology. Several of the results in this paper may be regarded as tools for thinning algorithms for images in Z 2 . We are working on analogous results for images in Z 3 .. Finally, we are also working on the design of new parallel thinning algorithms based on the previous construction of an (N , k)-retraction.

O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O


A sparse nonparametric hierarchical Bayesian approach towards inductive transfer for preference modeling In this paper, we present a novel methodology for preference learning based on the concept of inductive transfer. Specifically, we introduce a nonparametric hierarchical Bayesian multitask learning approach, based on the notion that human subjects may cluster together forming groups of individuals with similar preference rationale (but not identical preferences). Our approach is facilitated by the utilization of a Dirichlet process prior, which allows for the automatic inference of the most appropriate number of subject groups (clusters), as well as the employment of the automatic relevance determination (ARD) mechanism, giving rise to a sparse nature for our model, which significantly enhances its computational efficiency. We explore the efficacy of our novel approach by applying it to both a synthetic experiment and a real-world music recommendation application. As we show, our approach offers a significant enhancement in the effectiveness of knowledge transfer in statistical preference learning applications, being capable of correctly inferring the actual number of human subject groups in a modeled dataset, and limiting knowledge transfer only to subjects belonging to the same group (wherein knowledge transferability is more likely). Preference learning is a research field that has attracted significant attention by the machine learning community in the last years, as it provides the effective means for modeling and predicting a human subject's desires. Indeed, modern decision support systems (Chajewska, Koller, & Parr, 2000) , recommender systems (Blei, Ng, & Jordan, 2003) , and personalized service frameworks (Read, MacFarlane, & Cassey, 2002) heavily rely on sophisticated machine learning tools for preference modeling; computer games, e-commerce, marketing, and psychological support of patients are characteristic application areas that have greatly benefited from such methodologies, to name just a few (Yannakakis, Maragoudakis, & Hallam, 2009) .. Existing preference learning methodologies can be generally divided into two categories: (i) methodologies based on binary preference decisions; and (ii) methodologies based on instance scoring methods (Fürnkranz & Hüllermeier, 2003) . In the first case, the preference prediction problem is essentially formulated as an augmented binary classification problem, by utilizing some popular binary classification algorithm endowed with a suitable preference likelihood function (Aiolli & Sperduti, 200; Fürnkranz & Hüllermeier, 2003; Har-Peled, Roth, & Zimak, 2002; Herbrich, Graepel, Bollmann-Sdorra, & Obermayer, 1998) . On the other hand, under the second approach, a utility function is introduced to directly assign ranking scores to the presented alternatives, in a regression-type fashion (Chu & Ghahramani, 2005; Crammer & Singer, 2001) .. In this work, we follow the second approach: each alternative presented to a modeled subject is assigned a utility function, defined as the weighted sum of a set of basis functions, with each basis function expressing the dissimilarity of the presented alternative from a model training data point. Further, a suitable spherical Gaussian prior is imposed over the entailed weights vector of the model to conduct selection of the most informative training data points under an automatic relevance determination (ARD) scheme (Chatzis, Kosmopoulos, & Varvarigou, 2008) . In other words, a sparse Bayesian inference algorithm for preference learning is eventually obtained (Bishop & Tipping, 2000) . This formulation of the preference elicitation process under a sparse Bayesian learning paradigm enables our model to successfully deal with inconsistencies in the responses obtained by the modeled subjects during the model training phase, while is also very helpful in learning biases the subjects may have.. A significant issue preference learning methodologies are typically confronted with is the limited availability of labeled data suitable for model training. Typical machine learning approaches towards preference modeling consider each subject as an individual and unique case for which separate model training has to be 0957-4174/$ -see front matter Ó 2012 Elsevier Ltd. All rights reserved. doi:10.1016/j.eswa.2012.01.053 conducted, after acquiring some training data through interaction with that specific subject (Yannakakis et al., 2009 ). Indeed, obtaining appropriate model training datasets in preference learning applications requires significant time and effort from the modeled subjects. However, it is more than obvious that the average user of a real-world system would be rather reluctant to engage in a tedious training data collection procedure. Hence, in order for a preference learning methodology to cater to the actual needs of real-world systems, it must be capable of exploiting all the possibly available user information in the most efficient way, with the aim of minimizing the effort required from each individual subject for adequate model training.. To address these issues, in this work the task of learning the preferences of a modeled subject is conducted based on the assumption that each subject is not a unique and isolated case, but rather an instance or member of a group of subjects sharing a common underlying rationale in their way of making preference decisions. In other words, we seek to provide a model than can simultaneously handle the tasks of learning the preferences of multiple subjects, by regarding these tasks as a set of multiple correlated subtasks, instead of independent and isolated tasks. This way, under the proposed approach, the preference models of different subjects can regularize and influence each other, on the basis of the underlying assumption that the modeled subjects cluster into groups sharing common preference patterns.. Our approach comprises a typical example of a general machine learning framework usually referred to as multitask learning (MTL) (Caruana, 1997 ), or learning-to-learn (Thrun & Pratt, 1998 . Multitask learning has been the focus of much interest in the machine learning community over the last decade. The basic idea in multitask learning is that models learned on different scenarios have parts in common (Baxter, 1995; Evgeniou, Micchelli, & Pontil, 2005; Caruana, 1997; Lawrence & Platt, 2004) ; in a Bayesian framework this assumption is usually formulated as a hierarchical prior shared by the models pertaining to the considered individual scenarios (Yu, Schwaighofer, Tresp, Ma, & Zhang, 2003; Yu et al., 2004; Yu, Schwaighofer, & Tresp, 2005) . This hierarchical formulation of the MTL approach essentially consists of two layers: The bottom layer of the hierarchy comprises the individual, task-specific models. On the layer above, tasks are connected together via a common prior placed over the parameters of their corresponding models. Learning of the common prior is an indispensable part of the training process, and data from all tasks contribute to learning the common prior, thus enabling information transfer between tasks. On the other hand, given the common prior, individual models are learnt independently of each other. Consequently, learning of each task model is affected directly by its own training data, and indirectly by data from the other tasks (through the shared prior).. In this paper, we employ a hierarchical Bayesian formulation for our MTL construction, providing the flexibility to model both the individuality of each subject regarding their preference elicitation procedures, and the correlations between subjects (common preference patterns shared among subjects). Usually, the common prior in a hierarchical Bayesian MTL model is specified in a parametric form, e.g. as a Gaussian distribution with unknown mean and covariance; then, information is transferred between tasks by learning the hyperparameters of this prior using data from all tasks. This prior construction essentially reflects the assumption that all tasks are (equally) related to each other, sharing a common latent structure. However, this might not be quite the case in many real-life applications. Apparently, an ideal MTL algorithm should be able to automatically identify similarities between tasks and only allow similar tasks to share data or information. Indeed, in preference modeling applications, assuming that all subjects share some common rationale regarding their preferences is an oversimplistic and clearly unrealistic assumption. On the contrary, it is much more plausible to assume that the modeled subjects cluster into an unknown but learnable number of groups (subject clusters), with the subjects of each group sharing some common underlying rationale in their preference elicitation procedures.. In our work, we provide such a functionality by introducing the effective statistical tools that allow for the automatic identification of the similarities in the preference patterns between the various modeled subjects, and adjust accordingly the complexity of the model, that is, the number of groups the modeled subjects are clustered into. This is effected by adopting a nonparametric hierarchical Bayesian construction for our MTL preference model, relying on the consideration that the common prior shared among the modeled subjects takes the form of a Dirichlet process mixture (DPM) model. A similar construction was previously employed with significant success in Xue, Liao, Carin, and Krishnapuram (2007) for multitask learning of discriminative classifiers, allowing for the automatic identification of subgroups of related tasks, and providing more accurate classification predictions compared to simpler approaches such as single-task learning, pooling of data across all tasks, and other simplified MTL constructions.. DPMs are flexible Bayesian nonparametric models which have become very popular in statistics over the last few years, for performing nonparametric density estimation (Muller & Quintana, 2004; Neal, 2000; Walker, Damien, Laud, & Smith, 1999) . Briefly, a realization of a DPM can be seen as an infinite mixture of distributions with given parametric shape (e.g., Gaussian). This theory is based on the observation that an infinite number of component distributions in an ordinary finite mixture model (clustering model) tends on the limit to a Dirichlet process prior (Antoniak, 1974; Neal, 2000) . Indeed, although theoretically a DPM model has an infinite number of parameters, it turns out that inference for the model is possible, since only the parameters of a finite number of the mixture components need to be represented explicitly; this can be done by means of an elegant and computationally efficient truncated variational Bayesian approximation (Sethuraman, 1994) . Eventually, as a part of the model fitting procedure, the nonparametric Bayesian inference scheme induced by a DPM model yields a posterior distribution on the proper number of model component densities (inferred clusters) (Blei & Jordan, 2004) , rather than selecting a fixed number of mixture components. Hence, the obtained nonparametric Bayesian formulation eliminates the need of doing inference (or making arbitrary choices) on the number of mixture components (clusters) necessary to represent the modeled data.. The remainder of this paper is organized as follows: in Section 2, a brief overview of DPM models and probabilistic preference models is provided. In Section 3, the proposed sparse, nonparametric, hierarchical Bayesian approach towards inductive transfer for preference modeling is formulated, and an elegant truncated variational Bayesian inference algorithm for our model is derived. In Section 4, we thoroughly evaluate the efficacy of the proposed model through both synthetic experiments, and a real-world application to music recommendation. Finally, in the concluding section, we summarize and discuss our results. In this work, we introduced a novel sparse nonparametric hierarchical Bayesian approach towards inductive transfer for preference modeling. Our approach is based on the consideration of a hierarchical Bayesian formulation of the multitask learning framework, with the common prior shared among the tasks taken as a Dirichlet process mixture model with a special spherical Gaussian form for its component densities, which allows for the induction of a sparse nature for our model, by application of the ARD mechanism. This way, the proposed model conducts the data-driven discovery of groups of human subjects exhibiting some common underlying rationale in their preference elicitation procedures, and allows for information sharing among the human subjects of each group in the context of their model inference procedures. The efficacy of our novel approach was experimentally validated by considering both experiments with a synthetic dataset, and a real-world application dealing with automatic music recommendation.

O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O