Real-time synchronization of batch trajectories for on-line multivariate statistical process control using Dynamic Time Warping This paper addresses the real-time monitoring of batch processes with multiple different local time trajectories of variables measured during the process run. For Unfold Principal Component Analysis (U-PCA)-or Unfold Partial Least Squares (U-PLS)-based on-line monitoring of batch processes, batch runs need to be synchronized, not only to have the same time length, but also such that key events happen at the same time. An adaptation from Kassidas et al.'s approach [1] will be introduced to achieve the on-line synchronization of batch trajectories using the Dynamic Time Warping (DTW) algorithm. In the proposed adaptation, a new boundaries definition is presented for accurate on-line synchronization of an ongoing batch, together with a way to adapt mapping boundaries to batch length. A relaxed greedy strategy is introduced to avoid assessing the optimal path each time a new sample is available. The key advantages of the proposed strategy are its computational speed and accuracy for the batch process context. Data from realistic simulations of a fermentation process of the Saccharomyces cerevisae cultivation are used to illustrate the performance of the proposed strategy. Batch processes play an important role in several industries such as pharmaceutical, biochemical, food or biological, among others. The development of analysis and monitoring schemes for this kind of processes is an important task from an economical point of view. It is desirable when monitoring schemes are able to detect and diagnose disturbances efficiently in order to take actions on time to obtain consistent and good quality products in a consistent and profitable way. For this reason, the development of new methods based on the projection onto latent structures have experimented a great interest in the last years. These statistical methods are key tools for process understanding, multivariate statistical process control, abnormal situation detection, fault diagnosis and process control. All of these goals are included in modern improvement technologies such as Process Analytical Technology (PAT), fostered by United States Food and Drugs Administration (FDA) for the pharmaceutical industry.. Nomikos and MacGregor [2, 3] , in their pioneering work, introduced Batch Multivariate Statistical Process Control (BMSPC) based on Unfold Principal Component Analysis (U-PCA) and Unfold Partial Least Squares (U-PLS). Afterwards, other multiway methods were proposed to build empirical models for its subsequent application in monitoring batch process, such as Tucker models [4] and Parallel Factor Analysis (PARAFAC) [5] . One of the strong assumptions of most of these models is that all batch trajectories have Some authors emphasize the importance of using the warping information that comes out of the synchronization. Once raw data with unequal batch duration is aligned, the original time profile is removed from the data. Recovery of this primary information is essential to improve quality product prediction and also fault detection during a batch run. Westerhuis et al. [25] used the cumulative warp information as an extra variable in the U-PCA model in order to ease the fault detection. García-Muñoz et al. [8] used the cumulative time deviation from the average time as a new variable to reach a better fault detection, after synchronizing the batch data by the Indicator Variable approach.. Most of the research done on batch synchronization has been focused on off-line batch process monitoring. Unfortunately, there has not been a high interest to face the issue of batch synchronization in multivariate statistical process control for real-time applications. Only Kassidas et al. [1] and Ramaker et al. [24] comment on this topic. Nevertheless, there are several open issues deserving research. This is the main goal of this paper.. Section 2 introduces a brief review of the Dynamic Time Warping algorithm. A modification from Kassidas et al.'s approach [1] , based on introducing a semi-greedy search strategy, will be proposed in Section 3 to improve the alignment fit. Issues as the use of the warping information in the latent model, the strategies to give more importance to certain variables in the alignment procedure, some details about the inner DTW algorithm and the convenience of using warping information to improve the detection power of the model will also be discussed. In addition, some guidelines will be presented to carry out real-time synchronization for on-line multivariate statistical batch process control. Section 4 illustrates the performance of the proposed approach for real time alignment using simulated data from a Saccharomyces Cerevisae fermentation process. Finally, conclusions will be drawn in Section 5. This paper addresses the problem of unsynchronized batch trajectories for multivariate statistical process monitoring. As U-PCA-and U-PLS-based monitoring schemes assume that all batch trajectories have equal length, all process variables are measured at the same sampling frequency and key events happen at the same time, synchronization becomes a primary pre-processing step both in the modeling and in the exploitation phase.. Based on the original dynamic time warping (DTW) Kassidas et al.'s approach, a new time warping algorithm for on-line batch synchronization based on relaxed greedy strategy, called Relaxed-Greedy Time Warping (RGTW), is proposed in this paper. The new proposal avoids assessing the optimal path each time a new sample is available, increasing the computational speed. Furthermore, it reduces the uncertainty in the monitoring statistics and predictions in such a way that false alarms are notably reduced. The main contributions of the RGTW approach are: (1) a new global band constraint definition that takes into account the variability across all batches in each time period; (2) a definition of a cross-validated monitoring window in order to use an optimized window size yielding the minimum local optimal paths (a relaxed greedy strategy); and (3) a way to adapt mapping boundaries to the batch length. This last point is highly critical because in the on-line approach the endpoint of ongoing batch is unknown, hence, normal boundaries must be adapted by the current batch run.. Data from realistic simulations of a fermentation process of the Saccharomyces cerevisae cultivation are used to illustrate the good performance of the proposed strategy. Saccharomyces cerevisae is a yeast widely used in biotechnical and pharmaceutical industries for the production of proteins, so this yeast was chosen to simulate a fermentation process to illustrate the performance of the approach proposed in this paper. The model of the aerobic growth of S. cerevisae on glucose limited medium introduced by Lei [29] is used as a case study.. Fermentation is performed in batch mode where glucose is in excess in the medium, cells are not able to consume the whole amount of glucose, hence ethanol is produced together with the excretion of pyruvate and acetate. Later on, the initial amount of glucose is consumed by the growing cells. Before ethanol is consumed, the accumulated amount of pyruvate and acetate are consumed. During growth on ethanol, acetate is produced again. For more details about theoretical assumptions and additional information concerning the first principles model, readers are referred to [29] .. Process data was generated using Simulink for Matlab release 2007b R ( c The MathWorks, Inc). In order to simulate the physical uncertainty caused by the biological variability, slightly modified values of constants of the first principles model were introduced into the parametric space that defines the Simulink scheme for simulation. Also, Gaussian noise of low magnitude in the initial conditions (10%) and measurements (5%) to simulate the typical errors produced by sensors were added. Furthermore, the simulation achieved here have also taken into account the nature of this kind of processes. In fermentation processes, characterized by a duration of several days, some microorganisms may have different generation times, having a significant influence on biomass growth and quality features. Consequently, each batch performed has different durations.. A set of 50 unsynchronized batches was simulated under normal conditions and under the assumption commented previously. Ten variables were measured every sampling time over all batches: concentrations (glucose, pyruvate, acetaldehyde, acetate, ethanol and biomass), active cell material, acetaldehyde dehydrogenase (proportional to the measured activity), specific oxygen uptake rate and specific carbon dioxide evolution rate. Figure 3 (a) shows 4 variables (selected from the 10 process variables) for 50 good quality batches from the simulated process. The variables shown in the Figure 3(a) illustrates the lack of synchronization among the trajectories and also the dissimilarity in the batch length. The total length of the batches varied from 180 to 264 data points. Before carrying out the statistic modeling, the raw data need to be synchronized. In the selection of the batch which represents overall typical batch to batch variations, the closest batch to median length was taken as reference batch to carry out the synchronization by DTW. The median length was taken and not the average as statistical index to determine the reference batch because this statistic is less sensitive to outliers. In this case, the median length is 207 sampling times and the batch #4, whose batch length is 205, is the closest. All conditions and constraints described in section 2, except for the Sakoe and Chiba band constraint, were fixed. The reason why this global constraint was not established is because it is desirable to find optimal paths over all batches in order to get an overall envelope defined for all optimal pathways. As it is a preprocessing step characterized by alignment and process understanding, DTW algorithm is not totally constrained. An issue to be taken into account is which variables should be introduced into the DTW algorithm. It may happen that there are on/off variables providing information about different episodes or phases in the process that would be extremely useful in the synchronization step to improve the alignment. In contrast, these variables would not likely provide important information in MPCA modeling because they are not correlated with other variables. In this case, all variables measured were used to carry out the alignment. So Kassidas et al.'s and Ramaker et al.'s approaches were applied to calculate the weight matrix W, giving as a result (at 4th and 6th iteration, respectively) the weights shown in Figure 4 . Later on, the final weights used by the DTW algorithm were calculated as the geometric mean of the previous weights (see Figure 4) . After the DTW algorithm was applied to 50 NOC batches, a set of synchronized batches against batch #4 was reached (see Figure 3(b) ).. As explained in section 2, Kassidas et al.'s approach gives more weight to those variables more consistent from batch to batch, i.e. those variables whose trajectories are not notably deviating from the average trajectory. These variables are characterized by a high signal to noise ratio, hence high weights are given. In this case, the iterative procedure gave less weight to certain variables, such as ethanol and biomass concentration, active cell material and acetaldehyde dehydrogenase (from 5th to 8th variable marked as asterisks in Figure 4) , whereas the remaining variables were given more weight, being the 9th variable as the most weighted. This can be understood from Figure 3(b) where from the 6th to the 9th variables (biomass concentration, active cell material, acetaldehyde dehydrogenase and oxygen uptake rate) are plotted after synchronization. The first three plotted variables have a low signal to noise ratio in certain batch stages. In the 7th variable (active cell material), an important variation through the whole batch duration can be seen, therefore a low weight is expected. On the contrary, the 9th variable (oxygen uptake rate) has small differences between batch trajectories and the average trajectory, so a high weight is given. In an intermediate situation are the 6th and 8th variable, where larger differences are appreciated. In the case of the 6th variable (biomass concentration), from the beginning up to the 50th sampling time there is not much variation between the average trajectory and the test batches whereas from the 50th sampling time onwards, there is a considerable variation. Regarding the 8th variable, it seems to be consistent from the beginning to the 150th sampling time but after this point, a clear variation is detected. Hence, these two last variables are given an intermediate weight, being lower on the 6th variable than the 8th variable, as can be seen in Figure 4 . Since Ramaker et al.'s viewpoint, the weights should be given according to how much warping information is provided by a variable. The weights calculated by following this idea are drawn by circles in Figure 4 . This plot shows that variables with less warping information are the active cell material (#7) and the oxygen uptake rate (#9), whereas the variable acetaldehyde dehydrogenase (#8) has a high warping information content. Furthermore, it's worth emphasizing the fact that the 8th variable receives more weight following Ramaker et al.'s approach than Kassidas et al.'s approach. The reason for that is that in spite of its variability around average trajectory, the shape of the 8th variable's trajectory explains the main occurrences in the process represented by the majority of the fermentation variables, providing this way a better fit on the alignment. Our proposal, introduced in Section 2, is to average out geometrically the weights of both approaches in order to give weights more suitably. The idea is to give more weight to those variables more consistent from batch to batch and also providing important warping information. As can be seen in Figure 4 Before building the PCA model and carrying out the on-line monitoring of several batches using the RGTW algorithm, the width of the sliding window must be set and validated by the iterative procedure explained in Section 3. For that, from raw and synchronized batch data of the fermentation process, Pearson's correlation coefficients of warping information over all batches for a defined width were iteratively assessed, following the leave-one-out strategy. This allows us to determine the best window width that reduces the lack of fit caused by the greedy strategy. An Analysis of Variance (ANOVA) was performed on the Fisher Z-transformed correlation coefficients for the different window widths γ, showing statistically significant differences (p-value < 0.05). The Least Significant Difference (LSD) intervals displayed in Figure 5 show that correlation is statistically higher for window width γ = 4 (average r = 0.9999), whereas no statistically significant differences between window widths γ = 1 and γ = 2 were found. As a compromise between computational complexity and performance is desired in this case, a window width γ = 3 (average r = 0.9999) was selected. Figure 6 shows the warping information from all suboptimal paths found in the validation procedure drawn in Section 3. From the beginning up to the 50th sampling time, the warping information is smooth for all batches, meaning nearly no compressions or expansions were necessary to synchronize all batches. However, from the 50th to 120th sampling time, a greater number of vertical and horizontal transitions were required to align the batches. Specifically, a series of expansions (horizontal paths) can be observed between the 105th to 120th sampling time. One reason for this fact could be the phase change during the process. From this point there were an important number of compressions (vertical paths) because many batches have longer durations than the reference one.. All this warping information can be used to improve the detection of problems that may be masked by the synchronization. Nevertheless, special care must be taken when the warping information is taken into account in latent-based models. In processes characterized by large amount of variables, the inclusion of warping information as a new variable may cause this one to have no influence in the latent model. It is due to the fact that the warp may not be correlated with any of the process variables, thus this new variable would have little weight. In such cases, this variable should be considerably weighted in order to avoid this problem. In this work, the warping information will be introduced as a new variable into the model U-PCA without applying any special weight because of the features of this process. However, the capability of the model U-PCA to detect disturbances in the synchronization was checked as commented below. Once all batches were synchronized, they were arranged together with the warping information in a 50 × (11 × 205) [batches × (variables × time)] two-way matrix. Multivariate analysis based on U-PCA did not show any outlier. Three principal components were selected based on cross-validation results explaining 57% of total variation. Two monitoring charts, T 2 -Hotelling and SPE, were developed from the validated model, and also confidence limits were initially estimated from theoretical results [30, 31] . In order to achieve a suitable performance of the monitoring charts, control limits for the SPE chart are readjusted using data in a leave-one-out basis. Limits are raised or lowered so that the Overall Type I risk is equivalent to the imposed significance level (in this case at 95% and 99% confidence level for both statistics).. In order to illustrate the approach proposed in his paper, three simulated batches with abnormalities and a batch with good quality were simulated. The out-of-control batches were simulated by modifying the initial values of the constants k6, k10 and k11. Changing these constants, an abnormal behaviour on the formation of ethanol from acetaldehyde, the degradation rate of the active compartment (dependent on the glucose and ethanol concentration) and the degradation of acetaldehyde dehydrogenase compartment is expected, respectively.. The on-line synchronization approaches of Kassidas et al. and the one proposed in this paper are then used to achieve the real-time alignment of these simulated batches from the fermentation process. Both approaches are able to synchronize unequal batches but our method has some improvements over Kassidas et al.'s on-line implementation that will be shown by this example. Having built and validated both the PCA model and the parameters of the RGTW algorithm, the next step is to evaluate its ability to detect the out-of-control batches and time points and diagnose the cause of the out-of-control signal, i.e. the process variable(s) responsible for the abnormality. It is also worth knowing which approach produces more false alarms. For that, the NOC batches and three abnormal batches are synchronized and monitored in real-time. The results of the on-line synchronization from the start until the end of the batch run are summarised in Figure 7 . Suboptimal paths from the NOC batch and three abnormal batches assessed from the on-line synchronization, applying the RGTW algorithm (see Figure 7 (a)) and Kassidas et al.'s on-line approach (see Figure 7(b) ), are shown. The batches that present an abnormality on ethanol, active cell material and acetaldehyde are plotted with a yellow (bk6), pink (bk10) and red line (bk11), respectively. Kassidas et al's on-line DTW implementation needs to estimate the optimal path for each sampling time to achieve the synchronization. This causes the search algorithm of the optimal path to give different synchronization sequences as time goes by correcting decisions taken in the past. This can be observed from Figure 7 (a), where many branches come out from the root path. An immediate consequence will appear on the monitoring charts. The monitoring scheme will monitor the whole ongoing batch, from the start point up to the current sampling time, calculating the T 2 -Hotelling and SPE statistics every sampling time. Therefore, it is expected that the monitoring charts produce many false alarms (see Figure 8 ). This behaviour is not seen in the RGTW algorithm (see Figure 7 (b)) due to the fact that this algorithm uses a greedy strategy that provides a synchronized sample after the first γ steps. All optimal paths calculated for all synchronized batches are smooth, unlike the Kassidas et al.'s on-line implementation (see Figure 7(b) ). Hence, the most important advantages of using the RGTW algorithm are the computational time and memory save and the noteworthy reduction of false alarms. Another issue to discuss is the role of the warping information in the monitoring scheme. The result of the synchronization on the abnormal batches (see Figure 7) is different from the one on NOC batches (lightgreen paths). From the 25th up to the 60th and from the 70th up to 90th sampling time a different behaviour is detected in the warping information. During this time interval, there were more expansions (vertical paths) registered in the batch which presents an abnormality caused by a modification on the constant k11 (red path) than in NOC batches. Nevertheless, a different behaviour is presented in the abnormal batch which represents the degradation rate of the active cell material (the pink path). Unlike the abnormal batch (bk11), more compressions (horizontal paths) are shown from the 25th to the 60th sampling time than in the NOC batches. It seems the warping information may provide an extra information to detect the abnormality.. Once the result of the synchronization is obtained, Figure 9 shows the result of the on-line monitoring of the NOC batch and three abnormal batches through the whole batch run. SPE and T 2 -Hotelling charts are shown for all simulated batches with an abnormal behaviour and under NOC. It can be seen that the SPE control charts correctly signals the abnormal batches from the very beginning, since each simulated fault breaks down the correlation structure from the start point.. In Figure 10 , the SPE contribution plots at the time point 80 for the three faulty batches are shown. These charts allow diagnosing the particular fault. For the simulated batch with an abnormal behaviour on the formation of ethanol from acetaldehyde, the SPE contribution highlights the variables acetaldehyde (#3), acetate (#4), ethanol concentration (#5) and biomass concentration (#6) as responsible for the abnormality. Figure 10 (b) represents the variables responsible for the degradation rate of the active compartment, mainly, the active cell material (#7) and CO 2 evolution rate (#10). Finally, the monitoring scheme detected the Circles represent SPE statistic obtained at the sampling time t whereas, the asterisks represent SPE statistics assessed at the end of the batch after the synchronization was carried out. acetaldehyde (#3), acetate (#4) and biomass concentration (#6) to be the main variables responsible for the abnormality of the third faulty batch for the SPE statistic (Figure 10(c) ), while the ethanol concentration and the warping information received less weight. Another important issue is that for the three faulty batches the contributions plot gave at least some weight to the warping information variable. This shows that the monitoring scheme is able to detect an abnormal behaviour during the synchronization, as commented above. The proposed RGTW algorithm, as well as reducing false alarms produced by the synchronization, provides computation benefits compared to Kassidas et al.' s on-line implementation. In order to demonstrate the efficiency of the new approach, both algorithms were applied 1000 times to the NOC batch simulated, measuring the runtime each time. For that, the system clock on an Intel Core Duo with 4GB of RAM memory with minimal background processes running was used. At the end of the computation experiment, the average of runtimes measured was assessed. The RGTW algorithm was run by using three different window widths (γ=1,3,5) together with the bands defined by the cross-validation procedure. In the case of the Kassidas et al.' s implementation, Sakoe-Shiba global constraint with a width equal to M =59 (the difference between the largest length of NOC batches and the reference batch length) was set. The evaluation gave as a result that the RGTW algorithm was significantly faster than Kassidas et al.' s on-line DTW implementation for all window widths tested. Particularly, our approach required on average 0.241, 0.251 and 0.262 seconds to synchronize the NOC batch with a window width γ equal to 1, 3 and 5 units, respectively. In contrast, Kassidas et al.' s on-line implementation took 1.935 seconds on average. So, the maximum speed-up factor for this batch length is 8. An ANOVA was also performed on the run times obtained for each approach, showing statistical significative differences (p-value < 0.05 ) between RGTW and Kassidas et al.'s approach. When data are larger in terms of time points, differences on execution time will be greater. This is caused by the modifications proposed in this paper. Firstly, using the new global band constraint definition, the synchronization procedure is speeded up by a constant factor because less cells of the weight matrix W must be evaluated. Secondly, the relaxed greedy strategy avoids calculating the suboptimal path for each sampling time, thus the optimization problem is reduced to the window ζ instead of the whole batch length. Thirdly, the relaxed-greedy synchronization does not require the complete cumulative weighted local distance D be kept in the memory, as in the case of Kassidas et al.' s approach. It only needs to keep the current and γ − 1 previous columns in memory because the RGTW algorithm finds the suboptimal path within the window ζ instead of through the ongoing batch time.

O O O O O O O B I I I O B I I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


CXC Chemokines: Angiogenesis, Immunoangiostasis, and Metastases in Lung Cancer ABSTRACT: CXC chemokines have been found to be important in the regulation of angiogenesis and tumor-related immunity, and in promoting organ-specific metastases. This review highlights the importance of CXC chemokine ligands and receptors in mediating non-small cell lung cancer tumor-associated angiogenesis, "immunoangiostasis," and organ-specific metastases. These findings may ultimately lead to clinical strategies to target CXC chemokines in non-small cell lung cancer. Lung cancer is the leading cause of malignancy-related mortality. 1 The biology of lung cancer is complex, and a variety of biomarkers have been suggested to account for the virulence of lung cancer. Clearly, understanding the biology of factors that contribute to lung cancer tumorigenicity, avoidance of host immunity, and metastases will lead to novel strategies for therapeutic intervention of this devastating disease. In the last decade, studies have demonstrated that CXC chemokines/ chemokine receptors can directly enhance or inhibit tumor-associated angiogenesis, promote tumor-related immunity, and enhance organ-specific metastases. Although chemokine biology was originally felt to be restricted to only recruitment of populations of leukocytes, it has become increasingly clear that these cytokines can display pleiotropic effects in mediating biology that go beyond their originally described function. There is no better human disease to study diversity of chemokine function than tumor biology. Chemokines have autocrine, paracrine, and hormonal roles related to tumor growth and to metastasis to distant preferential organs. Moreover, CXC chemokines can play a critical role in mediating the full development of immunity to tumor-associated antigens. Understanding this expanded role of chemokines in tumor biology and using strategies to augment their expression to mediate immunoangiostasis will provide novel therapeutic intervention.

B I B O O O O O O O B I O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


On the Relation between Context-Free Grammars and Parsing Expression Grammars Context-Free Grammars (CFGs) and Parsing Expression Grammars (PEGs) have several similarities and a few differences in both their syntax and semantics, but they are usually presented through formalisms that hinder a proper comparison. In this paper we present a new formalism for CFGs that highlights the similarities and differences between them. The new formalism borrows from PEGs the use of parsing expressions and the recognition-based semantics. We show how one way of removing non-determinism from this formalism yields a formalism with the semantics of PEGs. We also prove, based on these new formalisms, how LL(1) grammars define the same language whether interpreted as CFGs or as PEGs, and also show how strong-LL(k), right-linear, and LLregular grammars have simple language-preserving translations from CFGs to PEGs. Once these classes of CFGs can be automatically translated to equivalent PEGs, we can reuse classic top-down grammars in PEG-based tools. Context-Free Grammars (CFGs) are the formalism of choice for describing the syntax of programming languages. A CFG describes a language as the set Email addresses: fabiom@dcc.ufrj.br (Fabio Mascarenhas), sergiomedeiros@ect.ufrn.br (Sérgio Medeiros), roberto@inf.puc-rio.br (Roberto Ierusalimschy) of strings generated from the grammar's initial symbol by a sequence of rewriting steps. CFGs do not, however, specify a method for efficiently recognizing whether an arbitrary string belongs to its language. In other words, a CFG does not specify how to parse the language, an essential operation for working with the language (in a compiler, for example). Another problem with CFGs is ambiguity, where a string can have more than one parse tree.. Parsing Expression Grammars (PEGs) [1] are an alternative formalism for describing a language's syntax. Unlike CFGs, PEGs are unambiguous by construction, and their standard semantics is based on recognizing strings instead of generating them. A PEG can be considered both the specification of a language and the specification of a top-down parser for that language.. The idea of using a formalism for specifying parsers is not new; PEGs are based on two formalisms first proposed in the early seventies, Top-Down Parsing Language (TDPL) [2] and Generalized TDPL (GTDPL) [3] . PEGs have in common with TDPL and GTDPL the notion of limited backtracking topdown parsing: the parser, when faced with several alternatives, will try them in a deterministic order (left to right), discarding remaining alternatives after one of them succeeds. Compared with the older formalisms, PEGs introduce a more expressive syntax, based on the syntax of regexes, and add syntactic predicates [4] , a form of unrestricted lookahead where the parser checks whether the rest of the input matches a parsing expression without consuming the input.. Ford [1] has already proven that PEGs can recognize any deterministic context-free language, but leaves open the question of the relation between context-free grammars and PEGs. In this paper, we argue that the similarities between CFGs and PEGs are deeper than usually thought, and how these similarities have been obscured by the way the two formalisms have been presented. PEGs, instead of a formalism completely unrelated to CFGs, can be seen as a natural outcome of removing the ambiguity of CFGs.. We start with a new semantics for CFGs, using the framework of natural semantics [5, 6] . The new semantics borrows the syntax of PEGs and is also based on recognizing strings. We make the source of the ambiguity of CFGs, their nondeterministic alternatives for each non-terminal, explicit in the semantics of our new non-deterministic choice operator. We then remove the non-determinism, and consequently the ambiguity, by adding explicit failure and ordered choice to the semantics, so now we can only use the second alternative in a choice if the first one fails. By that point, we only need to add the not syntactic predicate to arrive at an alternative semantics for PEGs (modulo syntactic sugar such as the repetition operator and the and syntactic predicate). We prove that our new semantics for both CFGs and PEGs are equivalent to the usual ones.. Our semantics for CFGs and PEGs make it clear that the defining characteristic that sets PEGs apart from CFGs is the ordered choice. For example, the grammar S → (aba | a) b is both a CFG and a PEG in our notation, but consumes the prefix ab out of the subject abac when interpreted as a CFG, and fails as a PEG.. We also show in this paper how our new semantics for CFGs gives us a way to translate some subsets of CFGs to PEGs that parse the same language. The idea is that, as the sole distinction between CFG and PEG semantics is in the choice operator, we will have a PEG that is equivalent to the CFG whenever we can make the PEG choose the correct alternative at each choice, either through reordering or with the help of syntactic predicates. We show transformations from CFGs to PEGs for three unambiguous subsets of CFGs: LL(1), Strong LL(k), and LL-regular.. A straightforward correspondence between LL(1) grammars and PEGs was already noted [7] , but never formally proven. The correspondence is that an LL(1) grammar describes the same language whether interpreted as a CFG or as a PEG. The intuition is that, if an LL(1) parser is able to choose an alternative with a single symbol of lookahead, then a PEG parser will fail for every alternative that is not the correct one. We prove that this intuition is correct if none of the alternatives in the CFG can generate the empty string, and also prove that a simple ordering of the alternatives suffices to hold the correspondence even if there are alternatives that can generate the empty string. In other words, any LL(1) grammar is already a PEG that parses the same language, modulo a reordering of the alternatives.. There is no such correspondence between strong-LL(k) grammars and PEGs, not even by imposing a specific order among the alternatives for each nonterminal. Nevertheless, we also prove that we can transform a strong-LL(k) grammar to a PEG, just by adding a predicate to each alternative of a nonterminal. We can either add a predicate to the beginning of each alternative, thus encoding the choice made by a strong-LL(k) parser in the grammar, or, more interestingly, add a predicate to the end of each alternative.. Our transformations lead to efficient parsers for LL (1) and strong-LL(k) grammars, even in PEG implementations that do not use memoization to guarantee O(n) performance, because the resulting PEGs only use backtracking to test the lookahead of each alternative, so their use of backtracking is equivalent to a top-down parser checking the next k symbols of lookahead against the lookahead values of each production.. There is no direct correspondence between LL-regular grammars and PEGs, either, given that strong-LL(k) grammars are a proper subset of LL-regular grammars. But we also show that we can transform any LL-regular grammar into a PEG that recognizes the same language: we first prove that right-linear grammars for languages with the prefix property, a property that is easy to achieve, have the same language whether interpreted as CFGs or as PEGs, then use this result to build lookahead expressions for the alternatives of each non-terminal based on which regular partition this alternative falls.. While LL(1) grammars are a proper subset of strong-LL(k) grammars, which are a proper subsets of LL-regular grammars, thus making the LL-regular transformation work on grammars belonging to these simpler classes, the simpler classes have more straightforward transformations which merit a separate treatment.. Given that these classes of top-down CFGs can be automatically translated into equivalent PEGs, we can reuse classic top-down grammars in PEG-based tools. So grammars written for tools such as ANTLR [4] could be reused in a parser tool that has PEGs as its backend language. As PEGs are composable, these grammars can then be used as components in larger grammars. The language designer can then start with a simple, LL(1) or strong-LL(k) subset of the language, and then grow it into the full language.. The rest of this paper is organized as follows: Section 2 presents our new semantics for CFGs and PEGs, showing how to arrive at the latter from the former, and proves their correctness. Section 3 shows how an LL(1) grammar describes the same language when interpreted as a PEG, and proves this correspondence. Section 4 shows how a simple transformation generates a PEG from any strong-LL(k) grammar, keeping the same general structure and describing the same language as the original grammar, and proves the latter assertion. Section 5 shows the equivalence between some right-linear CFGs and PEGs, and how this can be used to build a simple transformation that generates a PEG from an LL-regular grammar. Finally, Section 6 reviews related work, and Section 7 summarizes the paper's contributions and gives our final remarks. We presented a new formalism for context-free grammars that is based on recognizing (parts of) strings instead of generating them. We adopted a subset of the syntax of parsing expression grammars, and the notion of letting a grammar recognize just part of an input string, to purposefully get a definition for CFGs that is closer to PEGs, yet defines the same class of languages as traditional CFGs. These PE-CFGs define the same class of language as traditional CFGs, and simple transformations lets us get a PE-CFG from a CFG and vice-versa.. Our semantics for PE-CFGs has a non-deterministic choice operation. We showed how a deterministic choice operation based on the notions of failure and ordering turns PE-CFGs into quasi-PEGs; the addition of a not syntactic predicate then gave us a semantics that is equivalent to Ford's original semantics for PEGs. We then used our new formulations of CFGs and PEGs to study correspondences between four classes of CFGs and PEGs: LL(1), strong-LL(k), right-linear and LL-regular. We proved that LL(1) grammars already define the same language either interpreted as CFGs or as PEGs, as was already suspected, and gave transformations that yield equivalent PEGs for the grammars in the other three classes.. All our transformations preserve the structure of the original grammars; we do not change or remove non-terminals, nor change the alternatives of a nonterminal, just add predicates to the beginning or the end of each alternative. This means that our transformations have a practical application in reusing grammars made for one formalism with tools made for the other, as conserving the structure of the grammar makes it easier to carry semantic actions from one tool to another without modification.. Our transformations assume the presence of some kind of end-of-input marker, and incorporate this marker in the resulting PEG. This does not affect our equivalence results, but it does have implications in composability of PEGs resulting from our transformations. To use a PEG obtained from one of our transformations as part of a larger PEG (for embedding one language in another, for example) requires a suitable "end-of-input marker" to be picked (the boundaries between languages have to be explicit). We believe this problem should be easily solvable in practice.. A possible next step of this work would be the study of relationship between PEGs and classic bottom-up CFGs, such as LR(k) [26] and Simple LR(k) [27] . A key issue regarding this study is the fact that bottom-up grammars can have left-recursive rules, but a PEG with left-recursive rules is not complete [1] .. Recently, it was suggested a conservative extension of PEGs' semantics that gives meaning for left-recursive rules, so PEGs with these rules would also be complete [28] . Based on this PEGs' extesion we could try to establish a correspondence between bottom-up grammars and PEGs, and see if it is possible to achieve an equivalent PEG, with a similar structure, from a bottom-up CFG. Parsing Expression Grammars have generated much academic interest since their introduction by Ford [1] , with over sixty citations of Ford's paper in ACM's Digital Library. But just a few of these works are concerned with the theory of PEGs and their relation to other parsing tools; this section discusses these works and how they relate to our work.. Ford [1] leaves open the problem of how PEGs and CFGs relate, and does not outline a strategy to solve this problem. The solution of this problem for the major classes of top-down CFGs is a contribution of our work, and our recasting of the CFGs in a recognition-based formalism is another contribution that shows where CFGs and PEGs diverge. This paper is an extension of previous, unpublished work done [14] . This older work has the first version of the PE-CFG semantics, as well a the transformations between strong-LL(k) grammars and PEGs, and the proof of the correspondence between LL(1) grammars and PEGs. The proofs have been revised and improved.. Redziejowski [15] adapts the FIRST and FOLLOW relations of CFGs to the study of Parsing Expression Grammars, defining PEG analogs of these two relations. Redziejowski then uses the analogs for a conservative approximation of when an ordered choice is commutative or not, by defining a PEG analog of the LL(1) restriction. He admits that the approximation is too conservative for practical use, specially because of its treatment of syntactic predicates. We do not attempt to give definitions of FIRST and FOLLOW for PEGs, limiting our redefinitions of these relations just to our PE-CFG formalism, and using them to prove correspondences between LL(1) and strong-LL(k) CFGs and structurally similar PEGs, something that Redziejowski does not explore.. An earlier work by Redziejowski [16] presents several identities regarding the languages defined by PEGs, although the author concludes that the identities are only useful for obtaining approximations of a PEGs language, and he also does not try to relate the languages of PEGs and of CFGs.. In a more recent work [17] , Redziejowski, based on our earlier, unpublished work [14] , extends our correspondence between LL(1) grammars and PEGs to a larger class of grammars, and calls these grammars LL(1p). He notes that checking whether a grammar is LL(1p) is harder than checking if the grammar is LL(1). The paper also uses earlier versions of the new semantics for CFGs and PEGs that we presented in Section 2.. Schmitz [18] presents both an ambiguity detection algorithm for CFGs in the context of the SDF2 formalism, an extension of CFGs, and an algorithm for detecting whether an ordered choice in a PEG is commutative or not. Schmitz notes that an overly strict ambiguity detector can be as restrictive as introducing ordering, but does not attempt to further study the relation between CFGs and PEGs.. Parr and Quong [19] add semantic and syntactic predicates to LL(k) grammars to get the pred-LL(k) parsing strategy. The syntactic predicates of pred-LL(k) are only used in productions that have LL(k) conflicts. In these productions, the parser tries to match the predicates of each alternative in the order they are given in the grammar definition, choosing the first alternative with a predicate that succeeds. Backtracking is strictly local, as with PEGs, so a subsequent failure does not make the parser try other productions. The paper does not give a formal specification of pred-LL(k) grammars, nor how they relate to the class of LL(k) grammars.. Parr and Fisher [20] introduce the LL( * ) parsing strategy, which uses the basic idea of LL-regular grammars, with a predictive top-down parser for these grammars that uses a deterministic finite automata to select which alternative of a non-terminal to take. If the grammar is not LL-regular, the LL( * ) parser can use semantic and syntactic predicates with local backtracking, as in pred-LL(k), and also automatically introduce predicates, via a suitably named "PEG mode".. Generalized LL parsing [21, 22] extends the idea of LL(1) recursive descent parsing to the full class of context-free grammars (including ambiguous grammars), by making the parser proceed among the different conflicting alternatives "in parallel". It replaces the call stack of a recursive descent parser with a graph structured stack (GSS), a data structure adapted from Generalized LR parsers [23] . As the resulting parsers can parse ambiguous grammars, construction of the parse tree is non-trivial to implement efficiently [24, 21] , using a shared packed parse forest data structure also adapted from generalized bottomup techniques [25] .. Even if we restrict our domain to unambiguous grammars, we cannot use GLL parsing as a basis for a correspondence between CFGs and PEGs; a GLL parser is not a predictive parser, so we cannot try to encode the predictive part of the parser in a syntactic predicate, as we did for LL-regular grammars, nor we can exploit a property of the grammars, as we did for LL(1) and strong-LL(k) grammars.

O O O O B I O B I I B I O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Loops in canonical RNA pseudoknot structures Abstract. In this paper we compute the limit distributions of the numbers of hairpin-loops, interior-loops and bulges in k-noncrossing RNA structures. The latter are coarse grained RNA structures allowing for cross-serial interactions, subject to the constraint that there are at most k − 1 mutually crossing arcs in the diagram representation of the molecule. We prove central limit theorems by means of studying the corresponding bivariate generating functions. These generating functions are obtained by symbolic inflation of lv 5 k -shapes [11] . An RNA molecule is a sequence of the four nucleotides A, G, U, C together with the Watson-Crick (A-U, G-C) and U-G base pairing rules. The sequence of bases is called the primary structure of the RNA molecule. Two bases in the primary structure which are not adjacent may form hydrogen bonds following the Watson-Crick base pairing rules. Three decades ago Waterman et al. [9, 10, 13] analyzed RNA secondary structures. Secondary structures are coarse grained RNA contact structures. They can be represented as diagrams and planar graphs, see Fig. 1 . Diagrams are labeled graphs over the vertex set [n] = {1, . . . , n} with vertex degrees ≤ 1, represented by drawing its vertices on a horizontal line and its arcs (i, j) (i < j), in the upper half-plane, see Fig. 1 and Fig. 2 . Here, vertices and arcs correspond to the nucleotides A, G, U, C and WatsonCrick (A-U, G-C) and (U-G) base pairs, respectively. In a diagram two arcs (i 1 , j 1 ) and (i 2 , j 2 ) are called crossing if i 1 < i 2 < j 1 < j 2 holds. Accordingly, a k-crossing is a sequence of arcs (i 1 , j 1 ), . . . , (i k , j k ) such that i 1 < i 2 < · · · < i k < j 1 < j 2 < · · · < j k , see The length of an arc (i, j) is given by j − i, characterizing the minimal length of a hairpin loop. A stack of length τ is a sequence of "parallel" arcs of the form (1.1) ((i, j), (i + 1, j − 1), . . . , (i + (τ − 1), j − (τ − 1))), and we denote it by S τ i,j . We call an arc of length one a 1-arc. A k-noncrossing, τ -canonical RNA structure is a k-noncrossing diagram without 1-arcs, having a minimum stack-size of τ , see Fig. 2 . Let T k,τ (n) denote the set of k-noncrossing, τ -canonical RNA structures of length n and let T k,τ (n) denote their number.. We next introduce the following structural elements of k-noncrossing, τ -canonical RNA structures, see Fig. 3 and Fig. 4 . Let [i, j] denote an interval, i.e. a sequence of consecutive isolated vertices (i, i + 1, . . . , j − 1, j). We consider, see Fig. 4 (1) a hairpin-loop is a pair. (2) an interior-loop is a sequence Figure 3 . 3-noncrossing, 6-canonical structures: the pseudoknot structure of the PrPencoding mRNA represented as diagrams (top) and planer graphs (bottom)..

B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Model Reference Adaptive Control in Fuzzy-Based Context-Aware Middleware AbstractOwing to the dynamic characteristics of mobile environments, a mobile application needs For mobile applications to operate efficiently in mobile environments, they should be able to sense and evaluate the current operating context and then adapt their services to the context in order to optimize system performance. A generic context-aware middleware architecture-adaptive middleware infrastructure (AMI)-has been developed, which functions between the mobile applications and the operating system. The composite abstract contexts, which simplifies input factors for adaptation decisions. By incorporating control engineering techniques, the middleware also employs a model reference adaptive control mechanism to handle the dynamic aspects of adaptive control.. A fuzzy-based service adaptation model (FSAM) has been designed as the core inference engine in the middleware infrastructure. Our focus is on the service adaptation model, which compares context situations with the desired response from a model reference depository. Fuzzy linguistic variables were used to define context situations and the policies for adopting suitable services in the context-aware middleware. Fitness functions were designed to calculate the fitness degree for each service policy based on the distance between the service policy and the current context situation. The decision for service adaptation was made by selecting the policy with the best fitness degree with reference to a model reference depository. The experimental results have proved the effectiveness of the FSAM in our previous publications 12 . However, indepth experimental results also indicate certain limitations of the FSAM. When it comes to largegranular fluctuations, the FSAM cannot handle the adaptation decisions effectively with drastic changes in context situations. Control-theory and fuzzy-based approaches are combined to develop the Self-Adaptive Fuzzy-based Service Adaptation Model (SA-FSAM) in the middleware. By employing model reference adaptive control techniques in control engineering, the SA-FSAM monitors service adaptation in a real-time manner. The self-adaptive fuzzy-based approach is described in the following sections. In this research, a generic adaptive middleware infrastructure is introduced. The middleware infrastructure accommodates a self-adaptive model for service adaptation. A Model Reference Adaptive Control mechanism is implemented in the SA-FSAM middleware.. Control-theory and fuzzy-based approaches are combined to develop the SA-FSAM in the middleware. The core work lies on the Fitness Function and the Self-Adaptive Fitness Function. A campus assistant application is used to demonstrate the adaptation process. Based on the experimental results, it is observed that the SA-FSAM inference engine effectively alleviates the small-granular oscillations in context, and also smoothes the large-granular fluctuations that occur due to drastic changes in context information. The mechanisms behind the SA-FSAM are utilization of historical adaptation information and closed-loop control. With introduction of model reference adaptive control mechanisms, the static FSAM is enhanced to a dynamic control process. The performance of the context-aware middleware is enhanced in terms of resource utilization by avoiding drastic changes in adaptation decisions.. Certain limitations lie in the proposed framework. In practical scenarios, the research work provides a generic framework for mobile application development. However, the effectiveness also relies on the fuzzy membership functions used for defining context situations. It requires domain knowledge, user experience, and application preference to construct suitable membership functions for different context parameters. One direction of the future work would aim at developing suitable membership functions for different context situation parameters. Another direction of future research would be to enhance the adjustment mechanisms in the model reference adaptive control process by making use of the historical records of context values and services adaptations to make context predictions. Various researches on context-aware computing have been carried out in the past few years. In the Gaia Control theory, originally developed for handling industrial control and automation, has been applied to model physical systems. Control-theoretic approaches have been applied to computing systems with prevailing large-scale software systems and wide-area communication networks. The research community in software engineering domains and computing environments has established feedback loops as the core design elements in large-scale software development of adaptive systems 26 . Tu et al. 33 provided an implementation of load shedding by applying feedback control in a real-time data stream database system to balance the database workload. Robertson 29 proposed a closed-loop control mechanism for admission control in web server systems, and the experiments proved the stableness of the model. The major reason for using feedback was to reduce the effects of uncertainty in Internet load, which was difficult to model without feedback mechanisms. Fuzzy logic has been widely used since fuzzy set theory was first introduced by Zadeh 36 . A fuzzy logic controller 1, 17, 19 comprises of a fuzzification interface, a knowledge base that consists of a rule base and a data base, a decision-making logic unit, and a defuzzification interface. In mobile computing, researchers have been focusing on the use of fuzzy logic for mobile location management 7 and power control 8, 31 in wireless networks.. However, these systems were developed for adaptations relating to a single aspect. Recently, researchers have shown increasing interest on fuzzy controlled QoS adaptation 16, 25, 28 to fulfill application requirements in terms of more generic QoS specifications. In the CARISMA system, the context-aware middleware compared the application profile and the current context to evaluate which policy should be adopted 4, 5 . In the context-aware system implemented by Kwan et al. 23 , the functionality of a service code module was adapted on the basis of the estimated resource usage.. There has been a significant amount of research work on fuzzy-based adaptation. Ghinea et al. 16 developed a fuzzy logic-based descriptions of QoS specifications, as well as applying fuzzy logic to specifications based on user perceptions, and fuzzy programming was used to obtain a user-oriented ordering of the QoS parameters. Koliver et al. 22 proposed fuzzy rule-based techniques for assisting QoS adaptations in distributed multimedia systems. To deal with the uncertainty in variations in context environments, researchers have been building selfadaptive systems that are capable of dealing with continuously changing environments and emerging requirements by introducing feedback loop mechanisms in the design 9 . Li and Nahrstedt 24 introduced a fuzzy control model and a task control model to enhance the effectiveness of QoS adaptation decisions for a distributed visual tracking application. Applications were modeled as a serious of tasks assisted by a feedback loop. A middleware was used to implement the components to control adaptations so that various adaptive transient properties relating to stability and agility were addressed formally. These approaches were usually developed for specific problem domains (e.g., multimedia applications) while our SA-FSAM middleware emphasizes on a generic adaptation model that can be applied to different applications.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Locality preserving clustering for image database It is important and challenging to make the growing image repositories easy to search and browse. Image clustering is a technique that helps in several ways, including image data preprocessing, user interface designing, and search result representation. Spectral clustering method has been one of the most promising clustering methods in the last few years, because it can cluster data with complex structure, and the (near) global optimum is guaranteed. However, existing spectral clustering algorithms, like Normalized Cut, are difficult to handle data points out of training set. In this paper, we propose a clustering algorithm named Locality Preserving Clustering (LPC), which shares many of the data representation properties of nonlinear spectral method. Yet LPC provides an explicit mapping function which is defined everywhere, both on training data points and testing points. Experimental results show that LPC is more accurate than both "direct Kmeans" and "PCA + Kmeans". We also show that LPC produces in general comparable results with Normalized Cut, yet is more efficient than Normalized Cut. Image repositories are growing rapidly nowadays. It is important and challenging to make the repositories easy to search and browse. Image clustering is a technique that helps in several ways. For example, image clustering can be used as a preprocessing step that improves the speed and performance of content-based image retrieval (CBIR) [7] . For users who want to browse image database, hierarchical clusters of images will be useful for designed a convenient user interface (UI) [13] . Even in powerful search engines, image clustering will help to make more meaningful representation of query results [3] .. Image clustering is a technique that associates each image in database with a class label such that the images associated with the same label are similar to each other. Traditional clustering methods (such as Kmeans, Gaussian Mixture Model (GMM), etc) used in image clustering often get poor results in complex data, e.g. data points sampled from a non-linear manifold. Image database might be one of such examples because images with different concepts are generally not well-separated using traditional clustering methods. The reason why traditional methods failed is that the typical Gaussian distribution (or mixture of Gaussian distributions) is defined on the Euclidean space, and hence it can not always describe the data points sampled from a non-linear manifold. Note that, Parzen window [5] might be an exception, but it remains unclear how to apply it for clustering. In order to deal with such situations, spectral clustering method was proposed and has been successfully used in several applications. For example, Normalized Cut (NCut) [18] was used for image segmentation, video structuring [13] , scene detection [14] , video segmentation [16] and motion segmentation [17] . Ng et.al proposed a spectral clustering algorithm that extends classical Normalized Cut and gets better results [12] . Although Laplacian Eigenmaps [1] is focused on embedding of manifold data rather than clustering, it is easy to see that Eigenmaps is essentially equivalent to the embedding step of Normalized Cut in the case of certain kernel (e.g. Gaussian kernel).. In spite of the success of Normalized Cut and Eigenmaps on manifold data embedding and clustering, they can not provide us with an explicit mapping function. Actually, when dealing with new data points, similarities between the new points and all training data are needed [2] . The computation of the similarities can be very complicated due to the large size of training set. In order to solve this problem, Locality preserving projections (LPP) was proposed recently [8] and has bees used in document representation [9] and face recognition [10] . As a spectral embedding method, LPP shares many of the data representation properties of nonlinear methods such as Laplacian Eigenmaps. Yet LPP provides an explicit mapping function which is defined everywhere, either on training data points and testing points.. An interesting modification of spectral clustering is the out-of-sample extension [2] . Although the out-of-sample formula seems to be more useful for supervised learning than traditional clustering algorithm, it does help for spectral clustering. In fact, the state-of-the-art spectral clustering algorithms involve two steps, dimensionality reduction and traditional clustering like Kmeans. Out-of-sample extension is crucial for dimensionality reduction due to the saved computation. In detail, only the affini- Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ties between each data point and the points in a subset of the data, rather than the whole data set, are to be computed.. In this paper, we propose a clustering algorithm named Locality Preserving Clustering (LPC) which is a clustering method base on modified LPP. Technically, LPP solves a generalized eigenvalue problem, but the original LPP algorithm stated in [8] may mix true solutions (eigenvectors) with pseudo solutions and the trivial solution. We show that the pseudo solutions and the trivial solution carry no useful information and thus waste the dimensions of embedded results. In our version of LPP, these two kinds of useless solutions are well-handled, and a renormalization step is added to improve the robustness of the algorithm. Theoretic analysis is provided to show why modified LPP gives better and more robust results. In the algorithmic framework, we treat LPP as a feature selection method, and adopt a traditional clustering method (Kmeans in this paper) in the resultant space of LPP. We apply our methods on the problem of image clustering. Experimental results show that LPC is more accurate than both "direct Kmeans" and "PCA + Kmeans". We also show that LPC produces in general comparable results with Normalized Cut, yet is much faster than Normalized Cut.. It would be worthwhile to highlight several aspects of our proposed algorithm here:. LPC is defined everywhere, but traditional spectral clustering are only defined on training data points. This implies that clustering on incremental data points using LPC is much straightforward and faster than using traditional spectral clustering.. As a spectral method, the eigenvalue problem of LPC scales with the number of feature dimensions, while the eigenvalue problem of NCut scales with the number of data points. In most of image applications, the number of images in database is much larger than the number of feature dimensions. In this sense, LPC is much more efficient than NCut.. Since LPC is designed for preserving local structure, it is likely that the neighboring points in the low dimensional space are also near to each other in the original high dimensional space. So the clustering results of such data are probably more reasonable than that of data generated by PCA.. The rest of this paper is organized as follows: Section 2 describes the proposed Locality Preserving Clustering (LPC) algorithm which is based on modified LPP. Theoretic analysis of the modification is discussed in Section 3. The experimental results are shown in Section 4, followed by the conclusion in Section 5. [ ] In this paper we proposed a Locality Preserving Clustering (LPC) algorithm. LPC shares many of the data representation properties of nonlinear spectral method yet LPC provides an explicit mapping function which is defined everywhere, both on training data points and testing points. So LPC shows powerful capability for data representation and computational efficiency at the same time. We have designed three kinds of experiments and the results show the effectiveness of our proposed algorithm. Some other methods are implemented for comparison, i.e. Normalized Cut, PCA + Kmeans and direct Kmeans.

B I I O B I O O O O O O O O O B B O O B O B B O O O O O O O O O O B B O B I O O B O B B I B O O O O O O O O O O O O O O O O O B B O O B O O O B O O O O O B I O O O O O O O O B O O O B O O O B O O O O B O B I I O O O O O O B B O O O O B O O O O O O O O O O O O O B B O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O B B O O O O O O O O O O O O B O O B O B B O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O B B I O O O O O O O O O O O B I O O O O O O B O B O O O O O O O B O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O B O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B I B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O B B O O O O O O O O O O O O O O O O O O O B O O O O O O O B I I O O O O O O O O O O O B O O O O O O O O O O B O O O O O B B O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O B O O O O B O O O O O O O B O O O O B O B I I O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B I O O O O B B O O O O O O O B O O O O O O O O O O O B I O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O B O O O O O O O O O B I I O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B I I O B O O O O O B B O O O O B O O O O O O O O O O O O O B B O O O O O O O O O O B B O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O


Adaptive-Multilevel BDDC and its parallel implementation We combine the adaptive and multilevel approaches to the BDDC and formulate a method which allows an adaptive selection of constraints on each decomposition level. We also present a strategy for the solution of local eigenvalue problems in the adaptive algorithm using the LOBPCG method with a preconditioner based on standard components of the BDDC. The effectiveness of the method is illustrated on several engineering problems. It appears that the Adaptive-Multilevel BDDC algorithm is able to effectively detect troublesome parts on each decomposition level and improve convergence of the method. The developed open-source parallel implementation shows a good scalability as well as applicability to very large problems and core counts. The Balancing Domain Decomposition by Constraints (BDDC) was developed by Dohrmann [7] as a primal alternative to the Finite Element Tearing and Interconnecting -Dual, Primal (FETI-DP) by Farhat et al. [8] . Both methods use constraints to impose equality of new 'coarse' variables on substructure interfaces, such as values at substructure corners or weighted averages over edges and faces. Primal variants of the FETI-DP were also independently proposed by Cros [5] and by Fragakis and Papadrakakis [9] . It has been shown in [24, 38] that these methods are in fact the same as BDDC. Polylogarithmic condition number bounds for FETI-DP were first proved in [28] and generalized to the case of coefficient jumps between substructures in [15] . The same bounds were obtained for BDDC in [20, 21] . A proof that the eigenvalues of the preconditioned operators of both methods are actually the same except for the eigenvalues equal to one was given in [21] and then simplified in [3, 19, 24] . FETI-DP, and, equivalently, BDDC are quite robust. It can be proved that the condition number remains bounded even for large classes of subdomains with rough interfaces in 2D [13, 44] as well as in many cases of strong discontinuities of coefficients, including some configurations when the discontinuities cross substructure boundaries [29, 30] . However, the condition number does deteriorate in many situations of practical importance and an adaptive method is warranted.. Adaptive enrichment for BDDC and FETI-DP was proposed in [22, 23] , with the added coarse functions built from eigenproblems based on adjacent pairs of substructures in 2D formulated in terms of FETI-DP operators. The algorithm has been developed directly in terms of BDDC operators and extended to 3D by Mandel, Sousedík andŠístek [27, 35] , resulting in a much simplified formulation and implementation with global matrices, no explicit coarse problem, and getting much of its parallelism through the direct solver used to solve an auxiliary decoupled system. The only requirement for all these versions of the adaptive algorithms is that there is a sufficient number of corner constraints to prevent rigid body motions between any pair of adjacent substructures. This requirement has been recognized in other contexts [4, 18] , and in the context of BDDC by Dohrmann [7] , and recently byŠístek et al. [33] .. Moreover, solving the coarse problem exactly in the original BDDC method becomes a bottleneck as the number of unknowns and, in particular, the number of substructures gets too large. Since the coarse problem in BDDC, unlike in the FETI-DP, has the same structure as the original problem, it is straightforward to apply the method recursively to solve the coarse problem only approximately [7] . The original, two-level, BDDC has been extended into three-levels by Tu [41, 42] and into a general multilevel method by Mandel, Sousedík and Dohrmann [25, 26] . Recently the BDDC has been extended into three-level methods for mortar discretizations [12] , and into multiple levels for saddle point problems [37, 43] . The abstract condition number bounds deteriorate exponentially with increasing number of levels.. Here we combine the adaptive and multilevel approaches to the BDDC method in order to develop its variant that would preserve parallel scalability with an increasing number of subdomains and also show its excellent convergence properties. The adaptive method works as previously. It selects constraints associated with substructure faces, obtained from solution of local generalized eigenvalue problems for pairs of adjacent substructures, however this time on each decomposition level. Because of the multilevel approach, the coarse problems are treated explicitly (unlike in [27, 35] ). The numerical examples show that the heuristic eigenvalue-based estimates work reasonably well and that the adaptive approach can result on each decomposition level in the concentration of computational work in a small troublesome parts of the problem, which leads to a good convergence behavior. The developed open-source parallel implementation shows a good scalability as well as applicability to very large problems and core counts.. The theoretical part of this paper presents a part of the work from the thesis [36] in a shorter, self-contained way. Also some results by the serial implementation of the algorithm from [36] are reproduced here for comparisons. The two-dimensional version of the algorithm was described in conference proceedings [39] . The main original contribution of this paper is the description of the parallel implementation of the method, and numerical study of its performance.. The paper is organized as follows. In Section 2 we establish the notation and introduce problem settings and preliminaries. In Section 3 we recall the Multilevel BDDC originally introduced in [26] . In Section 4, we describe the adaptive two-level method in terms of the BDDC operators with an explicit coarse space. In Section 5 we discuss a preconditioner for LOBPCG used in the solution of the local generalized eigenvalue problems in the adaptive method. Section 6 contains an algorithm for the adaptive selection of components of the Multilevel BDDC preconditioner. Numerical results are presented in Section 8, and Section 9 contains summary and concluding remarks. We have presented the algorithm of Adaptive-Multilevel BDDC method for three-dimensional problems, and its parallel implementation. The algorithm represents the concluding step of combining ideas developed separately for adaptive selection of constraints in BDDC [23, 27, 31, 35] , and for the multilevel extension of the BDDC method [25, 26, 32] . The algorithm and its serial implementation was studied in the thesis [36] , and the serial algorithm for two dimensional problems also in [39] . The Adaptive BDDC method aims at numerically difficult problems, like those containing severe jumps in material coefficients within the computational domain. It recognizes troublesome parts of the interface by solving a generalized eigenvalue problem for each pair of adjacent subdomains which share a face. By dominant eigenvalues the method detects where constraints need to be concentrated in order to improve the coarse space, thus reducing number of iterations. On the other hand, the Multilevel BDDC aims at improving scalability of the BDDC method for very large numbers of subdomains, for which the coarse problem gets too large and/or fragmented to be solved by a parallel direct solver. However, as theory suggests and experiments confirm, Multilevel BDDC leads to an exponential growth of the condition number and the number of iterations.. The Adaptive-Multilevel BDDC method provides a kind of synergy of the adaptive and the multilevel approaches. Our results confirm, that adaptively generated constraints are capable of reducing the rate of growth of the condition number with levels. At the same time, the extension to three levels improved scalability of the adaptive 2-level approach, and for large problems and core counts, we have been able to obtain results we could not get by the 2-level method.. A convenient way of preconditioning the LOBPCG method based on components of BDDC was presented, effectively converting the generalized eigenvalue problems to ordinary eigenproblems. In our computations, this preconditioning led to large savings of number of LOBPCG iterations and corresponding computing time. However, to reduce the time necessary for solving the eigenproblems further, we have restricted the maximal number of adaptively generated constraints per face to ten in our parallel computations. For this reason, the resulting performance of the adaptive method was not as optimal as the serial tests (with arbitrary number of constraints per face) suggested.. We have described a parallel implementation of the algorithm available in our open-source library BDDCML. The solver has been successfully applied to systems of equations with over 400 million unknowns solved on 32 thousand cores. Presented results confirm, that both adaptive and non-adaptive implementations are reasonably scalable. However, presented computations have also revealed sub-optimal scaling especially on large numbers of cores, and these results will provide a basis for further optimization of the solver.. We have presented results of two benchmark and two engineering problems of structural analysis. On all problems, the adaptive selection of constraints led to reduced number of PCG iterations. However, for most problems, this fact did not lead to savings in computational time, and the cost of generating adaptive constraints was not compensated by saved iterations.. It can be concluded that the adaptive method is not suitable for simple problems, where also non-adaptive (even multilevel) method would converge reasonably fast. For problems with difficulties, the non-adaptive BDDC method leads to large cost of PCG iterations, especially for using several levels. On the contrary, the set-up phase with solution of local eigenproblems mostly dominates the overall solution time for the Adaptive-Multilevel BDDC method. Which approach is finally advantageous depends on a particular problem. Apart of the aspect of computational time, we have already encountered several problems for which the non-adaptive BDDC method failed, and which have been successfully solved by adaptive BDDC.

O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B