Mutual Information for Lucas-Kanade Tracking (MILK): An Inverse Compositional Formulation Abstract-Mutual Information (MI) is popular for registration via function optimization. This work proposes an inverse compositional formulation of MI for Levenberg-Marquardt optimization. This yields a constant Hessian, which may be precomputed. Speed improvements of 15 percent were obtained, with convergence accuracies similar those of the standard formulation. AN inverse compositional formulation for aligning a template and a reference image using mutual information is derived in this paper. The alignment or registration of a pair of images is an operation required in many applications such as image mosaicking [16] , simultaneous localization and tracking [6] , and multimodal image alignment [13] . In many applications, numerous registration operations are required. So, any improvements in the speed have a large effect on application performance as a whole.. Lucas and Kanade made one of the earliest practical attempts to efficiently align a template image to a reference image [9] , by minimizing the Sum of Squared Difference similarity function. Processing was limited by using a Newton-Raphson method to traverse the space of warp parameters. In Newton-Raphson optimization, iterative parameter updates to alignment parameters are obtained by multiplying the Jacobian by the inverse Hessian of the similarity function. Lucas and Kanade mainly considered translations, but they demonstrated that any linear transformation could be used.. Later research considered more complex transforms and attempted to reformulate the similarity function, allowing precomputation of some terms. In particular, Hager and Belhumeur [8] proposed inverting the roles of the reference and template at a strategic point in the derivation, and Shum and Szeliski [16] constructed the warp as a composition of two nested warps. In a general treatise on Lucas-Kanade (LK) techniques [1] , Baker and Matthews combined these methods to formulate the inversecompositional method.. Sum of Squared Differences (SSD) has several advantages as a similarly function: it is fast, it is simple to implement, it has a wide basin of convergence (making convergence easy), its gradient is simple to derive, and it is well understood. SSD's disadvantages include limited robustness to noise and variations in lighting conditions. Its wide basin of convergence can also make the result ambiguous. However, tracking multiple features and the use of models of appearance and structure can significantly improve robustness [5] .. Mutual Information (MI) is only slightly more expensive than SSD to compute and has several advantages. MI tolerates nonlinear relationships between the intensities in images and is robust to noise. MI has a sharp peak, giving a precise result. However, a starting point near to the solution is required. In the medical image registration field, MI is now widely used after its concurrent introduction and popularization by Viola and Wells [20] , Studholme et al. [17] , and Collignon et al. [4] .. Numerous MI implementations exist [13] , but few use an analytic derivative, limiting the optimization methods that may be used. The analytic derivative is difficult to obtain because of the nonlinear flooring functions implicit in the histogramming process used in calculating MI. Notable exceptions are an analytic derivative of MI using Partial Volume Interpolation by Maes et al. [10] ; and a derivative for MI using B-spline Parzen windowing by Thevenaz and Unser [18] . More recently, a general derivation of the four common types of MI was published by Dowson and Bowden [7] . The availability of a general analytic derivative for MI allows its use in the so-called Lucas-Kanade (LK) framework. This has implications to applications in both the computer vision and medical imaging communities.. The contribution of this work is to develop an inverse compositional formulation for MI. This uses two techniques: First, the alignment function is recomposed as a function of a base warp and a warp variation and, second, the roles of template and reference image are inverted or exchanged. This is difficult in the case of MI because the template and reference values are not separable into two terms. But, with some limited assumptions of constancy, speed-ups are still obtainable while maintaining the same accuracy as the conventional forwards-additive approach. Brooks and Arbel have also explored reformulating functions [3] . However, they use a BFGS optimizer, a bracketing and lineminimization method. BFGS only requires a Jacobian to be supplied and iteratively constructs a Hessian during optimization. In contrast, our approach evaluates a Hessian directly and uses this in a Levenberg-Marquardt (LM) algorithm, a Newton-type method. A direct comparison would really be considering two optimization philosophies rather than two formulations and is hence beyond the scope of this work.. The remainder of the paper is arranged as follows: After a background to image alignment in Section 2, the inverse compositional formulation of MI is presented in Section 3. The derivation obtained is compared to existing methods in terms of convergence and speed in Section 4 before the conclusions are given in Section 5. In this paper, an inverse compositional formulation for Mutual Information was introduced. This reformulates the MI function to yield an approximate Hessian that is dependent only upon the template image values and is therefore constant. The Hessian is approximate because cross terms between the reference and template intensities exist, which cannot be separated and vary with the warp parameter. These can be assumed to be constant too because the variation warp parameter is small. Inverse Compositional SSD on the other hand has an exact Hessian (to first order) because the effects of the two images are entirely separable.. The result of several experiments showed that inverse-compositional MI could compete with forwards-additive MI in terms of registration accuracy and demonstrated computational savings of up to 15 percent. This improvement occurred despite two confounding factors. First, Levenberg-Marquardt optimization does not require a Hessian to be evaluated every time a function is calculated. Second, the Hessian computation was optimized by reusing components utilized to calculate the function value. For methods where the Hessian is required every iteration, like the Newton method, the computational improvement would be greater. In testing, SSD and MI each performed best in different applications, suggesting that neither function is ideal in all circumstances.. The source code and test harness of this work have been made available at the authors' Web site. In future work, the effect of using more accurate joint-histogram approximations estimated from the marginal histograms will be investigated. Testing with nonrigid 3D data transformations will also be examined, as will reformulations of other functions like Normalized Correlation. To begin with a brief formalization of the registration process is required. Let I r represent a reference image and let I t represent a template image. The images are functions of 2D coordinate x 2 IR 2 . Some trivial changes to the formalization allow volumetric data IR 3 to be represented as well. Since I r and I t are represented as lattices of values at integral positions for x, interpolation is used to obtain values at nonintegral x values.. The registration process aims to locate the region in I r that most resembles I t by minimizing a distance function, f, which measures the similarity of the two regions. The position of I t relative to I r is specified by a warp function w with parameters v. v reg ¼ arg v min f½I r ðwðx; vÞÞ; I t ðxÞ: ð1Þ. The position of greatest similarity is found using an optimization method. f can be any similarity measure, e.g., SSD or MI. MI increases with greater similarity, so, to maintain convention, we minimize negative MI. For convenience and computational efficiency, I r is treated as infinite in extent and sampling to measure f is always performed within bounds of the defined region of I t . Regions outside the defined region of I r are defined as 0. Hence, I t is constant with respect to the warp parameters and computationally expensive boundary checking is avoided. Many optimization algorithms exist, but LK methods use a particular group of these: The so-called Newton-type methods, i.e., methods which assume locally parabolic topology and "jump" to the minimum using gradient information: v , and k indexes the iteration number. Newton methods should be contrasted with methods that choose a direction, bracket the minimum, and minimize along the line using Brent's algorithm [2] , e.g., Powell's Method or Variable Metric Methods [14] . Bracketing methods are more stable than Newton methods, but somewhat slower since more function evaluations are performed.. Minima in tracking and registration problems are frequently numerous and closely spaced, so the robustness of bracketing yields little advantage. Speed improvements, on the other hand, make multiple initializations practical, which can improve performance.. Generally, LK type methods apply Quasi-Newton optimization, i.e., an approximate Hessian,H, is used. In general, Newton and Quasi-Newton only perform well when near to the minimum. Steepest Descent methods, which ignore local curvature and instead multiply G by a scalar step-size value , perform better when further from the minimum. The Levenberg-Marquardt [11] method combines these two methods for optimal performance. A summary of these methods is supplied in Fig. 1 . Several experiments were undertaken to demonstrate the following:. .. Inverse-compositional MI converges as frequently and in the same number of iterations as forwards-compositional MI. .. Due the precomputation of the Hessian, the computational cost of registration is lower for Inverse-compositional MI than for forwards-additive MI. Experiments were performed using three image pairs for typical applications. These are shown in the first column of Fig. 2 . The first pair of images are two slices taken from a simulated MRI of a human brain using T1 and PD modes. Because the brain was simulated, the ground truth is known exactly. The different modalities also present difficulties for similarity that assume a linear intensity relationship between image pairs. In the second image pair, the template was extracted directly from the reference image of a natural scene. Although this is somewhat artificial, it allows the ground truth to be exactly known. The third example is of an indoor scene where the lighting conditions have changed substantially. The images were hand registered using key-points at a high resolution (2; 560 Â 1; 920) before being cropped and subsampled. Hence, the ground truth is known to within less than a pixel.. Simulated registrations using a six degree of freedom (DoF) affine warp from multiple initial starting points were performed. The initial positions were generated by randomly offsetting three of the corner points of the template and computing the parameters yielding the affine transformation between the ground truth and offset positions. This is similar to the test framework used by Baker and Matthews in [1] . In total, 600 tests per image were performed. These consisted of six groups of 100 tests, where a different standard deviation was associated with each group and a normal distribution was used for the random offsets. Standard deviations of 2, 4, 6, 8, 10, and 12 pixels were used. The results over 50 outerloop iterations of a Levenberg-Marquardt (LM) algorithm are plotted for each image in the second column of Fig. 2 . LM rather than Newton optimization was used since it's use of multiple function evaluations makes it more robust to tracking failure. The number of iterations was not fixed, two termination criteria were also used: when the change in function value was too low, i.e., jf ðkÞ À f ðkÀ1Þ j < 10 À4 ) and when the maximum change in any one warp parameter was too low, i.e., max n jÁv ðkÞ n j < 10 À4 , where n indexes the component of Áv.. The number of outer-loop iterations is not directly related to the number of function evaluations since the number of inner-loop iterations may vary. In LM optimization, G and H are only recalculated from the images once per outer-loop iteration. For inverse-compositional formulationsH is reused. Only f is reevaluated every inner loop iteration. Since LM optimization may terminate early, the number of inner and outer loop iterations is displayed, along with the mean time per optimization, in Fig. 3 . Additional optimizations were made, by reusing computational constructs utilized to calculate f, for calculating, G and again for calculatingH. This partially obscures the advantages of using a precomputedH since the cost of evaluatingH in addition to G is low. A larger increase in cost occurs when H is calculated using second order information. This is clearly shown in Fig. 4 , where the mean cost per evaluation per pixel for each method tested was measured over 20 tests.. In the first image, inverse-compositional MI managed to converge faster than forwards-additive MI. The difference arose because inverse-compositional MI utilizes an approximate Hessian as discussed in Section 3.1. The initially faster convergence was a surprising result which is believed to occur because the forwardsadditive approach updates the Hessian to model local conditions, leading it to take more conservative steps than the inversecompositional algorithm. However the constant histogram also has the side effect of a larger final error than the forwards-additive approach. The final error also increased with initial offset because the region in I r initially overlapped had less in common with the overlapped region when the algorithm finally converged. This implies that, once convergence is reached, optimization should be restarted with an updated Hessian that accounts for local conditions. An experiment to verify if restarting the algorithm would improve the final error for the inverse-compositional formulation was undertaken. One hundred random offsets with a standard deviation of 12 pixels were used used to initialize optimization for the medical image pair shown in Fig. 2a . The results of these tests, given in Fig. 5 , show that restarting the algorithm reduced the final error of the inverse-compositional method to that of the forwards-additive method. The re-evaluation of the Hessian caused by the restart came at an increased time cost of 2.9s versus 2.7s without the restart (3.3s for the forwards-additive approach). The SSD tests are of interest only in this case since the nonlinear relationships of multimodal medical images are well known to confound SSD.. In the second image pair, SSD converged faster than MI in every case. This is due to the direct match of the template and reference intensities (in the correct position) and the large basin of convergence typical of SSD. MI, on the other hand, appeared to become trapped or slowed by the local topology of function surface. Although MI is perhaps not the ideal distance function for the a problem like image 2, note how the inverse-compositional approach for MI performs as well as the forwards additive approach.. Image 3 is a difficult problem due to the radical changes to intensity induced by specularities on the object as lighting conditions change. MI's tolerance of nonlinear intensity relationships allows slightly better performance than SSD in that convergence is always toward ground truth, but convergence failure occurs in most cases. Inverse-compositional MI performs comparably to forwards-additive MI, although it too becomes trapped in local minima in many cases.. For MI, the mean time to convergence was faster for the inversecompositional formulation in most cases, despite the thriftiness of the LM method with Hessian evaluations. Time savings up to 15 percent were made. Approximately the same number of iterations were performed in both cases. For SSD, the inverse-compositional approach was almost twice the speed of the forwards-additive approach for image 2, agreeing with previous work [1] . The speed improvement was due partly to the efficiency of the inversecompositional formulation and partly to the fewer iterations required to converge on average. This pattern was not repeated for images 1 and 3 for two reasons: first, LM's thriftiness with Hessian evaluations and second, the cost of Hessian evaluation was reduced by reusing parts of the function and Jacobian evaluations.. The speed of SSD relative to MI indicated that, for many applications, it is still the method of choice. Standard sampled MI can compete with SSD in speed [7] , but, due to its noisy function surface, does not perform well for Newton-type optimization methods. It does perform well for random sampling optimization methods and with the simplex algorithm, but these algorithms do not use the Hessian, so an inverse-compositional approach holds no advantages. In addition to the tests above, a tracking algorithm was implemented using the strategic update approach of Matthews et al. [12] . In it, the tracking performance of forwards-additive MI and inverse-compositional MI were compared on a video sequence. For comparable real-time performance to SSD, standard sampled MI was used rather than the in-Parzen windowing approach used in the tests above. The results are shown in Fig. 6 . As shown, the inverse-compositional approach performed just as well as forwards additive. Results using SSD are also shown for interest. SSD also tracked quite well, but a large occlusion by the hand pulled the tracker off target.

B I O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Probability-based Dynamic Time Warping and Bag-of-Visual-and-Depth-Words for Human Gesture Recognition in RGB-D a b s t r a c t We present a methodology to address the problem of human gesture segmentation and recognition in video and depth image sequences. A Bag-of-Visual-and-Depth-Words (BoVDW) model is introduced as an extension of the Bag-of-Visual-Words (BoVW) model. State-of-the-art RGB and depth features, including a newly proposed depth descriptor, are analysed and combined in a late fusion form. The method is integrated in a Human Gesture Recognition pipeline, together with a novel probability-based Dynamic Time Warping (PDTW) algorithm which is used to perform prior segmentation of idle gestures. The proposed DTW variant uses samples of the same gesture category to build a Gaussian Mixture Model driven probabilistic model of that gesture class. Results of the whole Human Gesture Recognition pipeline in a public data set show better performance in comparison to both standard BoVW model and DTW approach. Nowadays, Human Gesture Recognition is one of the most challenging tasks in computer vision. Current methodologies have shown preliminary results on very simple scenarios, but they are still far from human performance. Due to the large number of potential applications involving Human Gesture Recognition in fields like surveillance (Hampapur et al., 2005) , sign language recognition (Starner and Pentland, 1995) , or clinical assistance (Pentland, 2005) among others, there is a large and active research community devoted to deal with this problem. Independently of the application field, the usual Human Gesture Recognition pipeline is mainly formed by two steps: gesture representation and gesture classification.. Regarding the gesture representation step, literature shows a variety of methods that have obtained successful results. Commonly applied in image retrieval or image classification scenarios, Bag-of-Visual-Words (BoVW) is one of the most used approaches. This methodology is an evolution of Bag-of-Words (BoW) (Lewis, 1998) representation, used in document analysis, where each document is represented using the frequency of appearance of each word in a dictionary. In the image domain, these words become visual elements of a certain visual vocabulary. First, each image is decomposed into a large set of patches, either using some type of spatial sampling (grids, sliding window, etc.) or detecting points with relevant properties (corners, salient regions, etc.) . Each patch is then described obtaining a numeric descriptor. A set of V representative visual words are selected by means of a clustering process over the descriptors. Once the visual vocabulary is defined, each new image can be represented by a global histogram containing the frequencies of visual words. Finally, this histogram can be used as input for any classification technique (i.e. k-Nearest Neighbor or SVM) (Csurka et al., 2004; Mirza-Mohammadi et al., 2009 ). In addition, extensions of BoW from still images to image sequences have been recently proposed in the context of human action recognition, defining Spatio-Temporal-Visual-Words (STVW) (Niebles et al., 2008) .. The release of the Microsoft Kinect TM sensor in late 2010 has allowed an easy and inexpensive access to almost synchronized range imaging with standard video data. Those data combine both sources into what is commonly named RGB-D images (RGB plus Depth) . This data fusion has reduced the burden of the first steps in many pipelines devoted to image or object segmentation, and opened new questions such as how these data can be effectively described and fused. Motivated by the information provided by depth maps, several 3-D descriptors have been recently developed (Bogdan et al., 2009; Rusu et al., 2010) (most of them based on codifying the distribution of normal vectors among regions in the 3D space), as well as their fusion with RGB data (Lai et al., 2011) and learning approaches for object recognition . This depth information has been particularly exploited for gesture recognition and human body segmentation and tracking. While some works focus on just the hand regions for performing gesture recognition (Wan et al., 2012; Li, 2012; Pedersoli et al., 2012; Biswas and Basu, 2011; Doliotis et al., 2011; Keskin et al., 2013) , in Shotton et al. (2011) introduced one of the greatest advances in the extraction of the human body pose using RGB-D, which is provided as part of the Kinect TM human recognition framework. The method is based on inferring pixel label probabilities through Random Forest from learned offsets of depth features. Then, mean shift is applied to estimate human joints and representing the body in skeletal form. Hernández-Vela et al. (2012) extended Shotton's work applying Graph-cuts to the pixel label probabilities obtained through Random Forest, in order to compute consistent segmentations in the spatio-temporal domain. Girshick et al. (2011) proposed later a different approach in which they directly regress the positions of the body joints, without the need of an intermediate pixel-wise body limb classification as in Shotton et al. (2011) . The extraction of body pose information opens the door to one of the most challenging problems nowadays, i.e. Human Gesture Recognition.. In the gesture classification step there exists a wide number of methods based on dynamic programming algorithms for both alignment and clustering of temporal series (Zhou et al., 2012) . Other probabilistic methods such as Hidden Markov Models (HMM) or Conditional Random Fields (CRF) have been commonly used in the literature (Starner and Pentland, 1995) . Nevertheless, one of the most common methods for Human Gesture Recognition is Dynamic Time Warping (DTW) (Reyes et al., 2011) , since it offers a simple yet effective temporal alignment between sequences of different lengths. However, the application of such methods to gesture detection in complex scenarios becomes a hard task due to the high variability of the environmental conditions among different domains. Some common problems are: wide range of human pose configurations, influence of background, continuity of human movements, spontaneity of human actions, speed, appearance of unexpected objects, illumination changes, partial occlusions, or different points of view, just to mention a few. These effects can cause dramatic changes in the description of a certain gesture, generating a great intra-class variability. In this sense, since usual DTW is applied between a sequence and a single pattern, it fails when taking into account such variability.. The problem of gesture recognition in which an idle or reference gesture is performed between gestures is addressed in this paper. In order to solve this problem, we introduce a continuous Human Gesture Recognition pipeline based on: First, a new feature representation by means of a Bag-of-Visual-and-Depth-Words (BoVDW) approach that takes profit of multi-modal RGB-D data to tackle the gesture representation step. The BoVDW is empowered by the combination of both RGB images and a new depth descriptor which takes into account the distribution of normal vectors with respect to the camera position, as well as the rotation with respect to the roll axis of the camera. Next, we propose the definition of an extension of DTW method to a probability-based framework in order to perform temporal gesture segmentation. In order to evaluate the presented approach, we compare the performances achieved with state-of-the-art RGB and depth feature descriptors separately, and combine them in a late fusion form. All these experiments are performed in the proposed framework using the public data set provided by the ChaLearn Gesture Challenge. 1 Results of the proposed BoVDW method show better performance using late fusion in comparison to early fusion and standard BoVW model. Moreover, our BoVDW approach outperforms the baseline methodology provided by the ChaLearn Gesture Recognition Challenge 2012. In the same way, the results obtained with the proposed PDTW outperform the ones from the classical DTW approach.. The BoVDW model for gesture recognition is introduced in Section 2, as well as the PDTW. Experimental results and their analysis are presented in Section 3. Finally, Section 4 concludes the paper. In this paper, the BoVDW approach for Human Gesture Recognition has been presented using multi-modal RGB-D images. A new depth descriptor VFHCRH has been proposed, which outperforms VFH. Moreover, the effect of the late fusion has been analysed for the combination of RGB and depth descriptors in the BoVDW, obtaining better performance in comparison to early fusion. In addition, a probabilistic-based DTW has been proposed to asses the temporal segmentation of gestures, where different samples of the same gesture category are used to build a Gaussian-based probabilistic model of the gesture in which possible deformations are implicitly encoded. In addition, to embed these models into the DTW framework, a soft-distance based on the posterior probability of the GMM was defined. In conclusion, a novel methodology for gesture detection has been presented, which is able to deal with multiple deformations in data.

O B I I O O O B I I O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


A Dynamic Geometry Reconstruction Technique for Mobile Devices Using Adaptive Checkerboard Recognition and Epipolar Geometry This paper describes a technique for reconstructing dynamic scene geometry using a handheld video projector-camera system and a single checkerboard image as a structured light pattern. The proposed technique automatically recognizes a dense checkerboard pattern under dynamic conditions. The pattern-recognition process is adaptive to different light conditions and an object's color, thereby avoiding the need to set threshold values manually for different objects when the scanning device is moving. We also propose a technique to find corresponding positions for the checkerboard pattern, when displayed by a projector, without needing any position-encoding techniques. The correspondence matching process is based on epipolar geometry, enabling the checkerboard pattern to be matched even if parts of it are occluded. By using a dense checkerboard pattern, we can construct a handheld projector-camera system that can acquire the geometry of objects in real time, and we have verified the feasibility of the proposed techniques. Recent advances in 3D scanning technology have made convenient and cost-effective systems possible. In particular, techniques using active structured light are well known for their accuracy and speed. However, during the past few decades, 3D scanning with these techniques has usually been aimed at static capture. Acquisition of 3D geometry in a dynamic environment remains an open research issue. Following the recent development of miniature projectors, ubiquitous projection that can present any surface as a display [4] , [14] is becoming practicable. Therefore, when realtime 3D scanning systems using such miniature projectors become more readily available, we will be able to utilize them for adaptive projection and the augmented reality applications around us [4] , [5] .. There are "one-shot" geometry-reconstruction techniques that can acquire the 3D shape of an object from a single captured image [25] , but they usually impose some conditions on the scanning surface, such as surface smoothness or constant color reflectance. If the conditions do not hold because of ambiguous colors or abrupt color changes caused by surface complexity or scanning device movement, the correspondence will be lost. Among the commonly used structured light patterns, a checkerboard pattern has not been used for geometry-acquisition purposes because it is difficult to encode the positional information in this pattern. Until now, checkerboard-recognition techniques have been proposed mainly for camera-calibration purposes [23] , [36] , where a checkerboard on a planar surface is used. Consequently, these techniques are usually sensitive to any distortion, occlusion, and noise in the pattern. This paper proposes a novel technique to recognize a dense black and white checkerboard pattern for one-shot geometry-acquisition purposes (Fig. 1) . The checkerboard pattern can be recognized automatically in dynamic conditions, with the internal edges being extracted adaptively. We exploit the directional properties of the feature points, the feature edges, and the graph topology to eliminate noise in the detected checkerboard pattern. This paper also presents a pattern-matching technique based on epipolar geometry that requires no position-encoding technique, with the corresponding positions being identified even if parts of the pattern are missing because of occlusion. This method enables the feature points to be extracted with high accuracy, and this enables us to easily find the corresponding position for each patch in a dense checkerboard pattern. Our experimental evaluation shows that the epipolar constraints of locally connected feature points are sufficient to solve for the correspondence robustly and effectively.. The remainder of the paper is organized as follows. The next section is about related work in checkerboardrecognition and shape-acquisition techniques. In Sect. 3, we present our method for recognizing the checkerboard pattern adaptively. This checkerboard-recognition technique Fig. 1 A handheld geometry acquisition system using a checkerboard pattern and a mobile projector-camera system.. Copyright c 2011 The Institute of Electronics, Information and Communication Engineers involves three steps, namely feature-point detection, featureedge detection, noise elimination and a labeling process. Section 4 describes a pattern-matching technique for the checkerboard pattern in which we show that the simple epipolar geometry of connected feature points and feature edges is sufficient to find the corresponding positions of segments of the checkerboard pattern. In Sect. 5, experimental results are discussed. Finally, the conclusion of the paper is given in Sect. 6. In this paper, we have proposed a one-shot 3D acquisition technique that uses a checkerboard pattern for structured light projection. The checkerboard pattern does not require any special position-encoding technique used by many existing studies. We have shown that, by using the local connectivity of simply categorized feature points and feature edges, the pattern can be analyzed robustly and is minimally affected by surface colors, discontinuity, or occlusion. The recognition technique is adaptive, and no manual setting of threshold values is required during the detection process, making this technique particularly suited to geometry acquisition in dynamic environments. Our checkerboard-recognition technique allows the recognition of dense and distorted patterns, which is particularly desirable for geometry-reconstruction systems. We have implemented a handheld scanning system by using a mobile projector and a Web camera to measure the geometry of a nearby object, and users can check the reconstructed geometry in real time.. In future work, we aim to merge 3D geometric data to create a 3D model in real time. It is an advantage of the handheld system that it can easily capture the geometry of an object from different angles. By merging the captured data, we can increase the accuracy of the model. We also plan to make the pattern adaptive to the environment by dynamically changing its horizontal and vertical line density. We believe that an adaptive pattern will produce a better 3D-scanned result in a dynamically changing environment.

O O B I O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


STRATEGIES TO SCAN PICTURES WITH AUTOMATA BASED ON WANG TILES Wang automata are devices for picture language recognition recently introduced by us, which characterize the class REC of recognizable picture languages. Thus, Wang automata are equivalent to tiling systems or online tessellation acceptors, and are based like Wang systems on labeled Wang tiles. The present work focus on scanning strategies, to prove that the ones Wang automata are based on are those following four kinds of movements: boustrophedonic, "L-like", "U-like", and spirals. Recent years saw a growing interest towards picture languages, and especially tilebased models. Picture languages are a generalization to two dimensions of classical string languages, where a picture is an array of symbols taken from a finite alphabet. Tiling problems have appeared in many branches of physics and mathematics like group theory, topology, quasicrystals, symbolic dynamics. More recently Winfree et al. [10] have demonstrated the feasibility of creating tiles made from folded DNA molecules that can act as Wang tiles [20] . As pointed out by Brun [7] , such models are self-assembling, like many biological systems, highly distributed, and parallel; they may be implemented using molecules, or a large computer network such as the Internet, thus opening several new prospectives. This idea is exploited in [8] , where an approach to the design of self-adaptive service-oriented applications based on a tilebased model is presented.. Among the various classes of languages defined by tile-based models, probably the most successful, as far as theoretical characterizations are concerned, is the class of tiling recognizable languages, also known as REC [12] . REC is a robust class, and coincides with the class of languages generated by various kind of devices, such as online tessellation acceptors [13] , tiling systems [11] , and Wang systems [9] .. Wang automata is a more recent kind of device characterizing REC, introduced by us in [18] , and based on the labeled variant of Wang tiles used in Wang systems [9] . Our model in some sense combine features of online tessellation acceptors, and traditional 4-ways automata [14] . Indeed, Wang automata are like online tessellation acceptors, because their computation assigns states to each input picture position; while the similarity with 4-way automata arises from the presence of an input head, which visits the input picture following a scanning strategy, i.e. a procedure to visit its pixels.. We originally introduced Wang automata to study the concept of determinism in REC. The concept of determinism for picture languages is not as straightforward as in string languages, where one may read the input string going from left to right or vice versa. The literature contains several different deterministic subclasses of REC, starting from deterministic online tessellation acceptors [13] , and going to the more recent [1, 4, 3, 17] . Indeed, Wang systems are implicitly nondeterministic: REC is not closed under complement, and the membership problem is NP-complete [15] . Wang automata overcome these issues by using a scanning strategy to move the input head: this allows us to introduce a natural and decidable notion of determinism, yielding a proper subclass of REC, called Scan-DREC, closed under complement, rotation and mirror operations. We refer the interested reader to [16] , where different subclasses of Scan-DREC determined by different scanning strategies, and their relation to unambiguity, are presented and studied.. In the present work, we focus on studying scanning strategies of Wang automata. In fact, we prove that polite scanning strategies, the ones Wang automata are based on, are essentially only those following four kinds of movements, and their rotations and symmetrical: boustrophedonic, where the head proceeds row-by-row, in a "snake"-like fashion; "L-like", where the head scans a row, a column, and then goes back; "U-like", where the head scans a column, a row, then another column, and goes back; and spirals. This result is interesting in two ways. First, it is now easier to study the properties of deterministic Wang automata, because we only need to focus on those few strategies. Second, the definition of Wang automata is now greatly simplified, because it does no more need all the theoretical scaffolding supporting generic kind of scanning strategies.. We consider now some related formalisms and results. Another kind of tile-based automaton, in some senses similar to Wang automata, is presented in [5] : quadrapolic automata are like to Wang automata, in that both use variant of labeled Wang tiles for working. Differently, quadrapolic automata do not read the input picture by following a fixed scanning strategy. To our knowledge, no definition of determinism for such devices is proposed in the literature.. As far as scanning strategies for picture languages are concerned, two relevant works are [2] and [6] .. In [2] , an automata model called tiling automaton is introduced, with the aim to define a general computational model for recognizable languages. This approach is centered upon the concept of scanning strategy itself, which directly depends on the size of the picture to be scanned. This definition is very general, and may exploit the size of the picture to perform "jumps", thus allowing complex behaviors. This freedom, together with the potential knowledge of the picture size, may be exploited to exceed REC.. In [6] , the considered strategies are "continuous", in the sense that the next considered position is adjacent to the current one. The actual definition of such strategies is presented in a qualitative form. This aspect could be source of some problems, since may admit different strategies depending on the picture size or shape (e.g. PeanoHilbert curves are suitable only for square pictures). Indeed, if we consider unary languages, scanning strategies which depends on the shape or size of the input picture may be exploited to exceed REC also in this case.. The paper is structured as follows. The first section introduces some notation and tiling recognizable languages. Section 2 defines Wang automata and their scanning strategies. Section 3 presents the main results on polite scanning strategies. Finally, the last section draws the conclusions. In this work we have proved that the behavior of the cyclic part of blind scanning strategies for Wang automata follows either a boustrophedonic, or "L-like", or "Ulike", or spiral pattern. The set of all possible scanning strategies is understandably finite, so we have also been able to mechanically check the main result through the bounded model checker Zot [19] , by expressing strategies as simple linear temporal logic formulae. 2 Clearly this fact permits a simplification of the model, because the actually usable strategies are simpler and more easily describable than the original generic ones presented in Section 2.1.. This result is also interesting from a theoretical point of view, since different scanning strategies define possibly different subclasses of REC, that we usually call µ-DREC, where µ is the chosen strategy. For instance in [16] , we proved that S-DREC and C-DREC are incomparable. Moreover, we know from [17] that the closure w.r.t. rotations of S-DREC is a remarkable class, because it coincides with the class of rowor column-unambiguous subclasses of REC defined in [1] . Therefore, the interesting and still open questions are about J-DREC, and U-DREC, and their relations with S-DREC and C-DREC.. As far as other possible future activities are concerned, one could extend the behavior of Wang automata by considering for instance scanning strategies which are "less blind", or based on a finite-state device, or permitting a bounded number of visits of the same position (i.e. not one-pass, but "bounded pass"). Clearly, many of these natural extensions could exceed REC, so they have to be handled with care.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Multi-frequency based location search algorithm of small electromagnetic inhomogeneities embedded in two-layered medium In this paper, we consider a problem for finding the locations of electromagnetic inhomogeneities completely embedded in homogeneous two layered medium. For this purpose, we present a filter function operated at several frequencies and design an algorithm for finding the locations of such inhomogeneities. It is based on the fact that the collected Multi-Static Response (MSR) matrix can be modeled via a rigorous asymptotic expansion formula of the scattering amplitude due to the presence of such inhomogeneities. In order to show the effectiveness, we compare the proposed algorithm with traditional MUltiple SIgnal Classification (MUSIC) algorithm and Kirchhoff migration. Various numerical results demonstrate that the proposed algorithm is robust with respect to random noise and yields more accurate location than the MUSIC algorithm and Kirchhoff migration. In the non-destructive evaluation area, an inverse problem is finding the specific characteristics (location, geometry, internal constitution etc.) of small inhomogeneities from measurements of scattered or far-field data. This problem, which arises in fields such as physics, engineering, and biomedical science, is highly related to everyday human life and is still a challenging problem [2, 6, 9] .. Among them, finding the locations of anti-personnel mines (embedded in the soil) from electromagnetic data (measured in the air) is an interesting and important problem in the military services. The mines have different material properties from the surrounding medium and they are small relative to the area. The main purpose of this kind of application is to find their locations more accurately, not to retrieve complete information. For this purpose, non-iterative MUSIC-type algorithm for finding locations of small inhomogeneities buried within a half-space at a fixed frequency has been developed, refer to [4] . MUSIC type algorithms are advantageous in the sense that they are fast, stable, can easily be extended to the multiple inhomogeneities, and they do not require specific regularization terms that are highly dependent on the problem at hand. However, due to the reason that one is faced with an aspect-limited inverse problem in the reflection mode because sources and receivers are located upper half-space, sometimes inexact locations are identified. Hence, in order to obtain an accurate locations of such inhomogeneities, an alternative algorithm that can overcome these problems is necessary.. Motivated by the above fact, we propose an effective, non-iterative location search algorithm at multiple frequencies that can work on limited view data in order to find the accurate locations of small electromagnetic inhomogeneities completely embedded within a homogeneous (lower) half-space. The starting point is that the collected MSR matrix can be approximated by a rigorous asymptotic expansion formula of the scattering amplitude in the presence of such inhomogeneities. This approximation leads us to proceed the singular value decomposition of the MSR matrix and to identify the structure of singular vectors. Applying the structure of singular vectors, a filter function operated at several frequencies can be designed for finding exact locations of inhomogeneities. Moreover, based on the statistical hypothesis testing, we can confirm that the proposed location search algorithm offers more exact information of locations than the established MUSIC-type algorithm and Kirchhoff migration.. Various numerical examples with noisy data will illustrate the behavior of the proposed location search algorithm operated at several frequencies. Unlike the numerical setting in [4] , computational examples under the narrow/wide range of incident and observation directions configuration and the closely located small inclusions situation show the feasibilities and limitations of the proposed algorithm. This paper is organized as follows. In section 2, we briefly survey the two-dimensional direct scattering problem for two-layered medium and introduce an asymptotic expansion formula for the scattering amplitude. In section 3, we design a location search algorithm by producing a filter function operated at several frequencies and compare its detection performance with traditional MUSIC-type algorithm and Kirchhoff migration. In section 4, corresponding numerical experiments with random noise are shown and compared with the result via MUSIC-type one and Kirchhoff migration to demonstrate its performance. In section 5 we give a brief conclusion.. Finally, we will refer to [13] as a useful reference that investigates the so-called reciprocity gap MUSIC as another algorithm linked to the MUSIC one. In this reference, the author shows that reconstructed positions of the small inhomogeneities via reciprocity gap MUSIC algorithm are slightly more accurate than the traditional MUSIC one. In this paper, we suggest a location search algorithm operated at several time-harmonic frequencies in order to find accurate locations of small electromagnetic inhomogeneities completely embedded within a homogeneous lower half-space. The approach is based on the asymptotic formulation due to the existence of small electromagnetic inhomogeneities. Throughout numerical simulations, we can conclude that the proposed algorithm not only performs quite well even in the existence of random noise but also successfully improves existing limitations (poor longitudinal resolution against an excellent transverse resolution) of MUSIC algorithm proposed in [4] and Kirchhoff migration. In addition, it still has some points of improvement for finding the locations of inhomogeneities under the situation ε + < ε − , µ + < µ − and wide range of incident/observation directions.. It is worth mentioning that such results obtained at low computational costs can be a good initial guess of a level-set evolution [1, 9, 11, 22] or of any other standard iterative algorithm. Although only two-dimensional problem have been considered herein, we expect that the proposed strategy, e.g., asymptotic formula, filter design, etc., could be extended to the three-dimensional problem, refer to [5, 15] for related works. Moreover, inconveniences of the proposed algorithm are also the same as those of SAR 1 . Hence, comparison of SAR and the proposed algorithm will be an interesting subject.. Finally, we would like to emphasize that the proposed algorithm can be extended to the shape identification of electromagnetically thin, arc-like, penetrable inhomogeneities, refer to [17, 18, 20] . Although, further mathematical investigation is necessary, we believe that it can also be extended to the identification of small or extended perfectly conducting cracks.

O O B I I O B I I O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Modulating ectopic gene expression levels by using retroviral vectors equipped with synthetic promoters Abstract The human cytomegalovirus and elongation factor 1a promoters are constitutive promoters commonly employed by mammalian expression vectors. These promoters generally produce high levels of expression in many types of cells and tissues. To generate a library of synthetic promoters capable of generating a range of low, intermediate, and high expression levels, the TATA and CAAT box elements of these promoters were mutated. Other promoter variants were also generated by random mutagenesis. Evaluation using plasmid vectors integrated at a single site in the genome revealed that these various synthetic promoters were capable of expression levels spanning a 40-fold range. Retroviral vectors were equipped with the synthetic promoters and evaluated for their ability to reproduce the graded expression demonstrated by plasmid integration. A vector with a self-inactivating long terminal repeat could neither reproduce the full range of expression levels nor produce stable expression. Using a second vector design, the different synthetic promoters enabled stable expression over a broad range of expression levels in different cell lines. Most commonly-used mammalian expression vectors employ strong constitutive promoters. These promoters include the immediate-early promoter of the human cytomegalovirus (CMV), the promoter of human elongation factor 1a (EF1a), and other viral promoters. Vectors equipped with these promoters are useful for ectopically expressing genes at high levels. Yet it can be important to evaluate low and intermediate gene expression levels, since different expression levels may generate different phenotypes. For example, high over-expression of the Ras oncogene can lead to premature cell-cycle arrest and senescence (Serrano et al. 1997) , while low over-expression can stimulate proliferation and induce tumor formation (Sarkisian et al. 2007 ). Additionally, in order to engineer and optimize genetic circuits or other synthetic devices in mammalian cells, it is paramount to be able to tune gene expression over a range of levels. To meet these needs, we set out to generate constitutive promoters that produce varying levels of expression.. The term ''synthetic promoter'' can refer to a constitutive promoter where the sequence has been modified so that it produces a different transcription level (Ruth and Glieder 2010; Jensen and Hammer 1998a) . Jensen et al. were among the first to report a library of synthetic bacterial promoters (Jensen and Hammer 1998b) . Since then libraries of yeast and mammalian promoters have been reported. In bacteria, synthetic promoter libraries have been employed to tune gene expression (Rud et al. 2006 ) and optimize lycopene production (Alper et al. 2005) . In yeast, synthetic promoter libraries have been used to vary gene expression (Hartner et al. 2008 ) and produce different glycerol yields (Nevoigt et al. 2006) . For mammalian cells, Tornoe et al. created a synthetic promoter library by generating chimeric promoters Electronic supplementary material The online version of this article (doi:10.1007/s11693-011-9089-0) contains supplementary material, which is available to authorized users.. assembled from SV40, beta-actin, and ubiquitin C sequences. The library was capable of a tenfold range in transcriptional activity (Tornoe et al. 2002) .. To generate synthetic promoters of varying expression strengths, one can mutate promoter sequences immediately upstream of the transcription start site (TSS). This region contains cis-regulatory elements that affect the frequency of transcription initiation and consequently the level of gene expression. In bacteria, researchers have successfully generated promoter variants by changing the bases near, but not within, conserved regulatory sequences, e.g., changing the spacing between the bases -35 and -10 to the TSS (Jensen and Hammer 1998b ). Yet in eukaryotic promoters, the regulatory sequences are not rigidly conserved and to generate mammalian promoters producing different levels of expression, one can vary consensus sequence elements. The TATA and CAAT boxes are two such elements found in promoters. TATA boxes have been identified in approximately 25% of human promoters (Xi et al. 2007 ). They are found approximately 30 bp upstream of the TSS and recruit the transcription pre-initiation complex and RNA polymerase. CAAT boxes are found in approximately 13% of human promoters (Xi et al. 2007) and bind the ubiquitous transcription factor NFY.. Here we report the generation of a library of synthetic promoters that can be used to evaluate and optimize ectopic gene expression over a range of levels. Synthetic promoters were generated by introducing mutations in and near TATA and CAAT sequences. To help ensure that the promoters would be compatible with vectors already used in the field, we generated variants of the commonly-used CMV and EF1a promoters. We demonstrated that retroviral vectors equipped with the synthetic promoters can be used to generate stable cell lines with desired levels of ectopic gene expression. We generated a library of synthetic promoters by mutating constitutive promoters commonly used in mammalian expression vectors. While the parent promoters normally produce high, possibly supra-physiological, expression levels, mutants demonstrated a range of low and intermediate levels. We also identified a retroviral vector construct that could produce desired expression levels in a generally predictable manner. The retroviral vector is equipped with antibiotic resistance for positive selection, a GFP reporter gene, and a cloning site for convenient recombinase-mediated insertion of genes of interest. Because the cloning site is compatible with other publically available cDNA libraries, e.g., that from the Human ORFeome project (Matsuyama and Yoshida 2009) , we envision that the vector will be immediately useful to many biologists and engineers. When using the promoters, one should bear in mind that expression levels can vary between cell types. While promoters derived from the same parent likely will preserve expression levels relative to themselves, users should still evaluate promoters in each cell type before proceeding with experiments. Users of our synthetic promoters should initially choose several promoters covering a wide expression range to test whether expression in their cells resembles that of the cell lines tested here (Fig. 4) . Whenever possible, we suggest using a reporter gene so expression levels can be continuously monitored. Vector components can affect the relative strength of the promoters and if users choose to deviate from our MLV vector design, we recommend including and evaluating a buffer region (e.g., another expressed gene as with our puromycin resistance cassette or a transcriptional insulator) upstream of the synthetic promoters. We believe that using our synthetic promoters to specify different expression levels can have advantages over an inducible expression system, e.g., expression using the tetracycline-responsive promoter. First, inducible promoters can be ''leaky'' in expression and even when switched ''off'' to the maximum extent, this uninduced level of expression may already be too high for one's purposes. Second, using inducible systems to express multiple genes requires an orthogonal system for each additional gene; i.e., one is limited by the number of available inducible systems. In contrast, when using different synthetic promoters to drive multiple genes at different expression levels, there is no such limitation. Third, when using inducible systems, each different expression level requires a different culture with a different inducer chemical concentration. In contrast, since the expression level from the synthetic promoters are independent of any supplemented chemical, multiple expression levels can be evaluated in a single culture. We envision transducing cells with a mixture of retroviral vectors equipped with the library of synthetic promoters. One can generate a single heterogeneous culture where a broad range of expression levels are represented. One could then use single-cell analysis, sorting (e.g., fluorescence-activated cell sorting), or enrichment methods (e.g., selection for proliferation) to determine a relationship between expression level and phenotype (Ferreira et al. 2011) . Finally, synthetic promoters can be useful in ''hard-wiring'' a genetic circuit or device so that it can function without chronic supplementation with inducer chemicals. We anticipate that the synthetic promoters and vectors reported here will be helpful to researchers seeking to evaluate genes at levels comparable to those in nature. They also will be useful in synthetic biology applications that require tuning of gene expression and system optimization.

O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


THE VALUE OF ROLLING-HORIZON POLICIES FOR RISK-AVERSE HYDRO-THERMAL PLANNING Abstract. We consider the optimal management of a hydro-thermal power system in the mid and long terms. From the optimization point of view, this amounts to a large-scale multistage stochastic linear program, often solved by combining sampling with decomposition algorithms, like stochastic dual dynamic programming. Such methodologies, however, may entail prohibitive computational time, especially when applied to a risk-averse formulation of the problem.We propose instead a risk-averse rolling-horizon policy that is nonanticipative, feasible, and time consistent. The policy is obtained by solving a sequence of multi-stage problems with deterministic constraints for the current time step and future chance and CVaR constraints.The considered hydro-thermal model takes into account losses resulting from run-of-river plants efficiencies as well as uncertain demand and streamflows. Constraints aim at satisfying demand while keeping reservoir levels above minzones almost surely. We show that if the problem uncertainty is represented by a periodic autoregressive stochastic process with lag one, then the probabilistic constraints can be computed explicitly. As a result, each one of the aforementioned multi-stage problems is an easy to solve medium-size linear program.For a real-life power system we compare our approach with three alternative policies. Namely, a robust nonrolling-horizon policy and two risk-neutral policies obtained by stochastic dual dynamic programming, implemented in nonrolling-and rolling-horizon modes, respectively. Our numerical assessment confirms the superiority of the risk-averse rolling-horizon policy that yields comparable average indicators, but with reduced volatility and with substantially less computational effort. The optimal operation of a hydrothermal system in the mid and long terms usually minimizes the expected value of the operating cost, essentially composed of fuel costs plus penalties for load shedding. This type of problem is of interest not only in centralized systems, but also for ISO and private and institutional agents acting on vertically integrated systems.. When the planning horizon covers several years, the optimal operation problem (OOP) is formulated as a stochastic program with recourse; see Birge and Louveaux (1997) . The problem is often large scale because there are many power plants and many time periods need to be considered. For a study over 5 years, a typical time step of one month -needed to describe suitably the hydrological uncertainty-makes up a total of 60 months. To eliminate "boundary" effects, see Section 5.1.1 below, in practice the time horizon is in fact doubled, yielding T = 120 stages.. For systems subject to cold winters, uncertainty in the seasonal demand needs to be represented too, increasing even more the problem complexity. For these reasons, a linear modeling is often adopted for the optimization problem.. A prototypical example is Brazil's power system, predominantly hydroelectric, for which the availability of limited amounts of hydro-power in the form of water stored in reservoirs makes the OOP very complex. There are many reservoirs in cascade, some of them with a capacity of regularization that covers several years, and spread over geographical regions with different seasonal rainfall. Water is a commodity of unknown value and uncertain availability and present operating decisions have future consequences that are difficult to quantify. In this setting, important indicators obtained when solving the OOP, like mean marginal prices or the average future unsupplied energy, need to reflect well the impact of extreme events such as extended droughts. This is why it is interesting to develop OOP variants that not only consider average phenomena, well represented by the cost in expected value, but also take into account the underlying risk factors. Since the hydrological risk is mostly associated to low volumes of stored water, operators struggle to keep the reservoirs over some critical minimum values or minzones that trigger an alert for the system as a whole, and require special actions to avoid future blackouts.. The question of how to measure risk in financial applications has been intensively developed over the past years, Artzner et al. (1999) , Rockafellar and Uryasev (2000) . In the energy sector, similar ideas have been applied for hedging financial risk of price-taker companies operating in To circumvent the curse of intractability and/or conservatism, inherent to multistage uncertain programs, we employ a risk-averse approach in a rolling-horizon setting, as in Guigues and Sagastizábal (2009), described in Section 2. Essentially, we consider solving successively T − 1 multi-stage problems with shorter and shorter horizons, for t = 1, . . . , T . For the t th multi-stage problem, constraints are considered deterministic at time step t, and in a probabilistic sense for. The approach builds feasible policies: all the constraints over the optimization period hold almost surely. This is a very important property for the OOP, for which the minzones constraints need to be satisfied with probability one. Moreover, the policy is time consistent in the sense of Shapiro (2009).. The considered hydro-thermal model considers uncertain streamflows and includes run-of-river plants, that can either spill or generate; see Section 3. Since turbines have a maximum capacity, some of the uncertain streamflow, meant to be immediately transformed into energy by the run-ofriver plants, may be lost. As explained in Section 3.1, the corresponding plant efficiency induces a loss of energy that depends on the whole system configuration and that is represented by a nondecreasing function of the streamflow. Constraints aim at satisfying demand while keeping reservoir levels above minzones almost surely. In Section 4 we show that if the streamflow is represented by a periodic autoregressive stochastic process with lag one, then the probabilistic constraints can be computed explicitly. As a result, each one of the aforementioned multi-stage problems is an easy to solve medium-size linear program. The model also takes into account uncertain demand. In Section 4.4 we address the general case, when demand is also a stochastic autoregressive process with lag one, for which calculations are no longer explicit, but involve estimations. A AND CLAUDIA SAGASTIZÁBAL For large-scale problems, rolling-horizon models are often impractical, because, as shown by Figure 4 , they need to generate scenario trees over each future step (t + 1, . . . , T ), at each time stage t. The corresponding calculations are prohibitive, making non-rolling horizon approaches more popular for stochastic programming problems with many time stages. The situation is only worse when building risk-averse policies, since there are more variables and constraints. We propose an alternative risk-averse policy, implemented in a rolling horizon that solves a sequence of chance-constrained problems. The explicit formulation of chance constraints for each full subtree makes our ra-RH policy not only feasible, but also "close" to the optimal one.. In particular, the results presented for the optimal operation problem of a large hydro-thermal power system show that our risk-averse rolling-horizon model is both realistic and tractable. Indeed, the model can efficiently handle energy losses arising when the turbines of run-of-river plants attain their maximum capacity and some streamflow is spilled. It is possible for operators to declare minzones, keeping reservoirs above pre-defined limits, with high probability. Similarly, the model can easily incorporate additional probabilistic constraints for flood control purposes, maintaining the reservoirs below a given threshold, if desired. On the basis of our numerical experience and for the case-study analyzed, the consideration of energy losses appears as an important differential factor. We believe such is the case in general for hydro-dominated power systems, especially if operators aim at keeping reservoir levels above critical trajectories.. We also analyzed the numerical behaviour of two commonly used alternative policies, obtained by applying either robust optimization in a nonrolling-horizon mode or stochastic dual dynamic programming, both in nonrolling-and rolling-horizons settings. With respect to robust optimization, for some problems, the objective function value at a robust solution remains close to the optimal value for small data perturbations. Such is the case of a collection of "bad" NETLIB problems considered in Ben-Tal and Nemirovski (2000), for which a maximum 0.1% data perturbation yields a robust objective function value distant in at most 1% from the optimal value. However, for a stochastic linear program like ours, with many time stages, streamflow realizations can exhibit a large variation between scenarios. As a result, uncertainty sets, calibrated for t = 1 and used to define robust counterparts, become just too large for far ahead time stages, degrading substantially the quality of the robust solution for some scenarios. This unfortunate feature is related to the Robust Optimization premise establishing that all realizations of the uncertain parameters should belong to the uncertainty sets considered in the robust counterpart. But in practice this is not the case, because calibrated uncertainty sets are mere confidence regions. When some scenario not covered by calibration appears in the simulation, the system gets to an unfeasible state, highly penalized. In our application, such is the case when there is load shedding: the large deficit cost gives an extremely high cost, and, in particular, produces unacceptable marginal prices.. With respect to the risk-neutral sddp-NRH and sddp-RH policies, our ra-RH approach gives close results in much less computational time. Moreover, our policy can be defined without resorting to scenario trees, nor having to use loose termination criteria to stop the iterations.. We finish with an important remark. From the above, one could conclude that a rolling horizon approach should systematically be preferred to other solution methods. In this respect, it is important to keep in mind that our comparisons are both problem and parameter dependent. The rolling-horizon approach has the potential weakness of producing unfeasible risk-averse problems, if confidence levels are too small. Such is not the case in our application; however, for a different type of problem, finding an a priori sound choice of confidence levels may be a challenging question, and other methodologies may be more suitable.

O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Computational protein biomarker prediction: a case study for prostate cancer Background: Recent technological advances in mass spectrometry pose challenges in computational mathematics and statistics to process the mass spectral data into predictive models with clinical and biological significance. We discuss several classification-based approaches to finding protein biomarker candidates using protein profiles obtained via mass spectrometry, and we assess their statistical significance. Our overall goal is to implicate peaks that have a high likelihood of being biologically linked to a given disease state, and thus to narrow the search for biomarker candidates.Thorough cross-validation studies and randomization tests are performed on a prostate cancer dataset with over 300 patients, obtained at the Eastern Virginia Medical School using SELDI-TOF mass spectrometry. We obtain average classification accuracies of 87% on a four-group classification problem using a two-stage linear SVM-based procedure and just 13 peaks, with other methods performing comparably.Modern feature selection and classification methods are powerful techniques for both the identification of biomarker candidates and the related problem of building predictive models from protein mass spectrometric profiles. Cross-validation and randomization are essential tools that must be performed carefully in order not to bias the results unfairly. However, only a biological validation and identification of the underlying proteins will ultimately confirm the actual value and power of any computational predictions. Recent advances in mass spectrometry (MS) technology are starting to enable high-throughput profiling of the protein content of complex samples. It is foreseeable that MS, coupled with chromatographic separation techniques, might become complementary to microarray technology on the proteome level. The very dynamic nature of the proteome, the wide range of abundances, the general lack of a "protein catalog" (unlike the genomic catalog, which is all but complete for a number of organisms) and various technical challenges in capturing proteins make this a particularly ambitious and challenging undertaking. While MS has been used extensively on purified, digested samples to identify proteins via peptide mass fingerprints, the data we use in this paper are fundamentally different since they consist of mass spectra (or, more precisely, peaks from mass spectra) of complex mixtures such as blood serum. After some chromatographic separation steps (which are crucially important, but not the primary subject of this paper), a mass spectrum of the matrix-crystallized sample is obtained on a wide mass range (in our case, 2-40 kDa) in order to obtain a profile of the protein content of a sample. If reproducibility is ensured, then these spectra can be used to identify peaks whose intensities correlate with a particular phenotype of interest, e.g., in this paper, prostate cancer.. The purpose of this paper is to show that computational methods can be useful in narrowing the search for protein biomarker candidates. Once we find a small set of peaks that can be used to computationally "predict" phenotypes with high accuracy, these peaks should be analyzed further and the underlying proteins identified, e.g, by focusing an MS/MS instrument on the relevant peak masses. The hope is that the subsequent functional study of these proteins will eventually lead to new biological insights into disease pathways and, ultimately, to reliable diagnostic tests and potential therapeutic targets. We want to stress the need for biological validation; the inherent variability of mass spectrometry data makes it uncertain whether peak profiles can be used for diagnosis directly.. The need for computational methods is evident in order to find peaks that correlate with phenotypes and, equally importantly, in order to assess their statistical significance. We survey several classification (or supervised learning) methods that can be used in this context and apply them to a multi-class prostate cancer prediction problem. Given that sample sizes for these kinds of experiments are typically small, and given that validation of any results we produce requires laborious protein identification, our aim is to find the smallest set of peaks that yields reasonable (i.e., statistically significant) classification results.. The literature on classification techniques as well as the related field of machine learning is vast; we will not even attempt to give a summary but will rather refer the readers to, e.g., the excellent book by Hastie, Tibshirani and Friedman [1] . Several of these methods have been utilized for discriminating cancer samples from normal (control) samples using proteomic data. The two main components of these approaches are 1) the feature selection (dimension reduction) method and 2) a classification method to build a predictive model. For example, Petricoin et al. [2] used genetic algorithms for feature selection and Kohonen's self-organizing map classification in their study of ovarian cancer (see also Sorace and Zhan [3] for a more detailed and principled analysis of the same data). Li et al. [4] used the signal-to-noise ratio for an initial feature selection and, subsequently, used "unified maximum separability analysis" repeatedly for classification in their breast cancer study. Adam et al. [5] and Qu et al. [6] used the "area under the Receiver Operating Characteristic (ROC) curve" criterion for feature selection and decision trees in conjunction with boosting techniques for classification in a prostate cancer study. Lilien et al. [7] analyzed the same data and the ovarian cancer data from Petricoin et. al. [2] with a probabilistic algorithm based on principal components and linear discriminant analyses.. In this paper we contrast the performance of several selected statistical and optimization-based techniques on the SELDI-TOF data from Adam et al. [5] and, in contrast to previous studies, assess their prediction significance via randomization techniques. These methods were selected for this study based on their simplicity and their widespread use and availability. While this is far from a comprehensive benchmark study, we make several points about the need for rigorous cross-validation and randomization.. By dividing the data set at our disposal into training set (to be used for model building) and test set (to estimate the generalization power of the model), and by doing this randomly and many times over, we can benchmark various classification methods reliably and gain insights into their capabilities of handling proteomic data. In particular, we used Fisher's linear and quadratic discriminant functions, nonparametric kernels, nearest neighbor methods and linear support vector machines for classification. Misclassification rates are biased (downward) when both the training and test sets are used for feature selection as opposed to when only the training set is used. While performing the cross-validation studies, care was taken so that the test set does not influence the choice of the peaks used in the classification.. Although we will use the above mentioned prostate cancer data for illustration, the data analysis strategies and the methods used here are of general applicability and could easily be adapted for other mass-spectrometry datasets. In earlier work [8] we have proposed candidate biomarkers in lung cancer using protein profiles obtained via MALDI-TOF (matrix assisted laser desorption and ionization-time of flight) mass spectra (data provided by Duke University's department of Radiology) by employing similar data analysis strategies and classification methods. One of the five peaks found to be most useful in classifying lung cancer, and the only one up-regulated in cancer, was subsequently identified by Howard et. al [9] as stemming from Serum Amyloid A, an inflammatory marker, which we take to be an encouraging indication that this kind of analysis indeed has the potential to reveal relevant biomarkers.

O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O