An atlas-based method to compensate for brain shift: Preliminary results Compensating for intraoperative brain shift using computational models has shown promising results. Since computational time is an important factor during neurosurgery, a priori knowledge of the possible sources of deformation can increase the accuracy of modelupdated image-guided systems. In this paper, a strategy to compensate for distributed loading conditions in the brain such as brain sag, volume changes due to drug reactions, and brain swelling due to edema is presented. An atlas of model deformations based on these complex loading conditions is computed preoperatively and used with a constrained linear inverse model to predict the intraoperative distributed brain shift. This relatively simple inverse finite-element approach is investigated within the context of a series of phantom experiments, two in vivo cases, and a simulation study. Preliminary results indicate that the approach recaptured on average 93% of surface shift for the simulation, phantom, and in vivo experiments. With respect to subsurface shift, comparisons were only made with simulation and phantom experiments and demonstrated an ability to recapture 85% of the shift. This translates to a remaining surface and subsurface shift error of 0.7 ± 0.3 mm, and 1.0 ± 0.4 mm, respectively, for deformations on the order of 1 cm. Ever since its advent, medical imaging has played a significant role in surgical planning and treatment because it provides valuable information about anatomical structures and function. This has been particularly helpful for neurosurgical procedures where often the surgeon has to remove a tumor without damaging the healthy brain tissue surrounding it. In order to take advantage of image guidance during a neurosurgical procedure (also known as imageguided neurosurgery, IGNS), preoperative tomograms of the patient must be registered to the patient's anatomy in physical space. While image-to-patient rigid alignment is relatively straight-forward, clinical studies in IGNS have exposed limitations to this approach. Systematic studies have reported that the brain is capable of deforming during surgery for a variety of reasons, including pharmacologic responses, gravity, edema, surgical manipulation and respiration Nimsky et al., 2001; Nabavi et al., 2001) and that the brain can shift a centimeter or more in a non-rigid fashion (Hartkens et al., 2003) .. To correct for deformations, various imaging techniques such as computed tomography (CT) (Butler et al., 1998) , magnetic resonance imaging (MRI) , and ultrasound (US) (Letteboer et al., 2005) have been investigated for intraoperative image-guided surgery. CT procedures have been questioned for their dose exposure, while MR procedures are considered cumbersome and have been questioned for their cost-effectiveness. Current US systems suffer from low soft-tissue contrast and lack image clarity as compared to CT and MR imaging methods. Therefore, in their current state intraoperative imaging systems do not present a complete solution for guidance correction of the brain shift phenomenon.. As a cost-effective and efficient method, computational modeling is a procedure that can translate complex surgical events into accurate estimates of tissue response and thereby compensate for intraoperative brain shift. In model-updated image-guided neurosurgery (MUIGNS), a biomechanical model of brain shift is driven with sparse data 1 to accurately deform preoperative images to their current intra-operative position. Several groups have investigated the potential value of physical/biomechanical models underpinned by various biomechanical concepts (Miga, 1998; Wittek et al., 2005; Miller, 1999; Skrinjar et al., 2002; Edwards et al., 1998; Hagemann et al., 1999; Ferrant et al., 2001) . Towards this end, Paulsen et al. (1999) reported a 3-D biomechanical model governed by consolidation mechanics. Additional development of the equations and their solutions can be found elsewhere Miga, 1998) . In this work, a patient-specific mesh is created and case-specific boundary data such as tumor resection and/or tissue retraction is imposed to generate updates of the preoperative images over the entire course of surgery. Despite previous success with the model approach Platenik et al., 2002; Miga et al., 1998 , there are several remaining challenges discussed below that need addressing in order to attain the goal of MUIGNS.. One of the greatest challenges presented by MUIGNS is that the computational time associated with the model does not meet the real-time constraints of neurosurgery. For a 23,000 node, 123,500 element mesh with four degrees of freedom (DOF) at each node on a single central processing unit (CPU) Silicon Graphics Indigo workstation, reported a run time of 8.5 min to simulate gravity-induced deformations, 6.5 min to simulate retraction, 5.5 min to simulate excision and 6.0 min to simulate unretraction for the first time step, and 5.75-7.0 min for every subsequent time step. These run times can be significantly improved with use of parallel processing and more powerful computers, as demonstrated by Warfield et al. (2002) . Using a Sun Microsystems Sun Fire 6800 with 12 750 MHz UltraSPARC-III CPUs, for a 43,584 nodes, 214,035 element mesh with three DOF at each node, Warfield et al. reported a run time of 15 s. These performance improvements are encouraging and will only add to the impetus to bring complex models to the operating room.. Another critical component of MUIGNS is the accurate translation of boundary conditions during the course of surgery. For example, the amount of cerebrospinal fluid (CSF) loss during surgery and the head orientation of the patient in the operating room (OR) may be two important factors in determining the degree of shift from gravitational forces . Although the preoperative surgical plan can provide an estimate of the patient's orientation a priori, estimates for the degree of change in buoyancy forces acting on the brain are somewhat more elusive. Related to the hydrated nature of the brain, intracranial pressure from the edematous tissue surround tumors can cause the brain to swell within the craniotomy region. Models that are biphasic in nature may be better suited to capture these brain shift effects. In addition, deformations from retractor blades and internal strain energy changes that occur during tumor exposure and resection can also contribute to brain shift during surgery. Each of these factors present a challenge with respect to prescribing boundary and internal forcing conditions. While nonguided prediction is desirable, there is little doubt that the accuracy of brain shift models can be increased by integrating feedback from sparse intraoperative data . These sparse displacement measurements can be obtained from a number of sources (Nauta, 1994; Hartkens et al., 2003; Lunn et al., 2001; Sun et al., 2003 Sun et al., , 2005 Sinha et al., 2005) . Sparse intraoperative data is typically used as displacement or stress boundary conditions to constrain the computational model. Using the measured sparse displacements, Ferrant et al. (2002) and Skrinjar et al. (2002) rigidly constrained their computational model to exactly match the measured displacements, as if they were known boundary conditions. Though this method is relatively easy to implement, it faces the potential limitation that since boundary displacements are constrained to match measured surface displacements, artificial forces can be introduced at the measured points which by observation are stress-free. More specifically, in this framework, all deformations result from the application of contact forces when the force environment is considerably more complex and could involve a myriad of contact and distributed loading conditions. Given this, it is worth exploring alternative approaches for integrating sparse intraoperative data with computational modeling. Also, it should be noted that time taken to compute the displacements using the computational model and the time taken to integrate the sparse data with the computational model, must meet the real-time constraints of neurosurgery. Thus the incorporation of sparse measurements must not only improve accuracy, but also significantly reduce the time required to update the preoperative images.. In recent developments, Lunn et al. (2005) presented a novel method that corrects brain shift by combining a best prior estimate (BPE) with a force perturbation correction technique to better match sparse data to model output. The reported method casts the model correction with a nonlinear optimization framework which uses the method of Lagrange multipliers to rapidly correct their BPE of brain deformations. They call the method the adjoint equation method (AEM) and have had encouraging preliminary results. While the mathematical approach is quite elegant, it still represents a challenging optimization framework that is significantly under-determined. Also in addition to the computational cost, the AEM reduces modeling efforts to solve for the optimal distribution of forcing functions rather than concentrating on generating a more deterministic model. For the sole purpose of shift correction, this is quite appropriate but it focuses the framework at using models to regularize data rather than model deformation events.. In the work presented here, a constrained linear inverse model is combined with a biomechanical tissue model to best fit the measured sparse intraoperative data. Initially presented in Dumpuri et al. (2003) , the method reported here extends the earlier framework by incorporating a smoothing constraint to improve the efficiency and accuracy of solution. In order to account for the degree of uncertainty associated with all the sources of deformation, the computational model is run multiple times and these multiple model solutions are combined with the help of a inverse model to predict the intraoperative brain shift. It should also be noted that a considerable amount of the framework can be pre-computed and that at this time the inverse approach is a direct solution. With this technique, the model solutions act as training samples for the inverse model and the sparse intraoperative data act as control points, thereby removing the degree of uncertainty associated with MUIGNS. The framework proposed herein has some specific distinctions from the work of others: (1) the atlas of deformations is constructed from simulations based on physiological events, therefore the framework moves beyond the role of image interpolator to one that provides quantitative estimates of deformation-related properties (e.g. stresses, interstitial pressure dynamics, etc.), (2) the atlas of solutions generated is of more considerable breadth and attempts to include all the forces causing intraoperative brain shift and the varying surgical presentations of the patient (e.g. mannitol induced deformations, gravity-induced sag, and resection), (3) the inverse model is linear, and takes advantage of pre-processing, (4) the framework introduces a simple weighting scheme to constrain the atlas, and (5) presents a semi-automatic boundary condition generator to translate the boundary conditions encountered in the OR and should allow for the easy reproduction by others.. In this study, the fidelity of a constrained linear inverse model approach is demonstrated in a phantom experiment, two in vivo cases and a simulation study. It should be noted that though the sparse intraoperative data can include both pressure and displacement measurements, displacement data was chosen to test the accuracy of the proposed inverse model. In this study a laser-range scanner (LRS) is used to acquire sparse data measurements (Sinha et al., , 2006 . The laser-range scanner used in Sinha et al. (2005) is capable of generating a three-dimensional point cloud corresponding to (x, y, z) cartesian coordinates and two-dimensional texture co-ordinates (u, v) . In Sinha et al. (2005 Sinha et al. ( , 2006 ) the LRS was modified by the attachment of 12 infrared light emitting diode (IRED) markers, allowing for the scanner to be tracked in physical-space. The approach to measuring brain shift using LRS is as follows: LRS is used to scan the cortical surface, the initial scan is registered to the patient's preoperative images thereby establishing a correspondence between image-space and physical-space , the brain then deforms during surgery, and LRS is used to acquire a serial scanning dataset of the cortical surface after deformation. The shift acquired is then transformed to physical-space coordinates with the aid of a calibration phantom. The shift-tracking protocol using LRS has been described in detail in Sinha et al. (2005) . These sparse intraoperative measurements are used to constrain the linear inverse model. Also to meet the real-time demands of neurosurgery a parallel implementation of the computational model on a multiprocessor architecture is considered. In is interesting to note early reports dismissing methodologies to correct for intraoperative shift that did not involve traditional intraoperative imaging (specifically, iMR, and iCT) (Rubino et al., 2000) . These early reports believed that conditions such as swelling and brain volume changes due to hyperosmotic drugs could not be predicted or practically modeled. In the experiences shown here and by others, these conclusions continue to be challenged and the potential for computer modeling within the OR environment is only now being realized. There is a growing acceptance that predicting brain shift at scales relevant to surgical interventions through computer models is very possible when proper approximations to forcing conditions are understood and when sufficient data is present to guide predictions. Albeit for surface displacements, the results presented in this work show that with a good set of basis-solutions/training-samples, the constrained linear inverse model can be used to predict cortical shift. In future work, the accuracy of this approach in predicting full volume displacement fields from sparse intraoperative data sets will be achieved using a comprehensive digitization approach. Further approach enhancements are being pursued and will include more anatomical constraint information and possibly a non-linear optimization framework. The integration of sparse intraoperative data into MUIGNS is not a trivial task. As stated in , sparse intraoperative data applied in an interpolative/extrapolative sense cannot capture the entire range of deformation. They also note that the sparse intraoperative displacement data must be applied in a manner that is consistent with the forces causing those displacements. The constrained linear inverse modeling approach proposed here achieves this integration in an efficient manner. Although it may seem time consuming to build an atlas of deformations, results from Section 4.1 indicate that using a multiprocessor environment significantly reduces the amount of time taken to generate atlases. As stated in Section 4.1, using 16 processors it takes approximately 117 s to calculate one basis solution/training sample in the deformation atlas. Therefore for a deformation atlas with 320 basis solutions, using 16 processors it required approximately 10 h to build each atlas reported in Section 3.2. It should be noted that a sensitivity analysis has not been performed which may indicate that the level of detail in the atlases presented here may not be necessary to achieve meaningful shift corrections during surgery. The results here are encouraging given this relatively modest atlas; and perhaps similar results may be achieved with sparser training sets. This awaits further study.. The in vivo cases reported in this work were treated as unknown systems, i.e. the surgeon did not generate the PPoE but rather retrospective estimates based on operation notes were used. Ultimately, the PPoE will be provided by the surgeon using an ordinary graphical user interface (GUI) one day prior to the surgery. The information provided will reflect the anticipated patient orientation, craniotomy size, and location of the brain stem in reference to the preoperative image volume. Once these have been designated, the automatic BC generation is performed to sample the possible deviations from the PPoE. This boundary condition atlas and the model is then submitted to the multi-processor cluster which returns a deformation atlas several hours later. This strategy has several distinct advantages: (1) it accounts for the uncertainty in distributed surgical loads, such as the gravitational sag and the physiological parameters like the amount of mannitol that will be administered, in a realtime sense (2) the method relies on relatively inexpensive small-scale computer clusters, (3) the time-consuming calculations are performed preoperatively, and (4) all forms of data (e.g. fMR, PET, SPECT, etc.) can be mapped within each solution and either combined through the inverse model or they can be actively computed based on the displacement fields predicted using the constrained linear inverse model. It should also be noted that the compensation for distributed loading conditions is only the first stage in this compensation strategy. The second stage is to monitor the more direct interactions such as retraction, and resection. These actions are more representative of surface loading conditions as opposed to distributed ones. We hypothesize that direct predictive modeling approaches with these should deliver the required accuracy. Previous experience with animal systems supports this tenet (Platenik et al., 2002; . Surface loadings resulting from tissue retraction and the resulting Mean ± standard deviation (maximum) of the measured/total shift have been reported. a Six different displacement sets were used to constrain and test the fidelity of constrained linear inverse model. Therefore average maximum total shift and the standard deviation of the shift over the six different displacement sets has been reported. deformations can be modeled as a multistep process ) and this has been demonstrated previously (Platenik et al., 2002; . This paper represents an approach to the more difficult distributed loading conditions.. The results from the Phantom experiments are important on two distinct levels. The first level is in validating the model approximation reported by Miga et al. for modeling brain sag , i.e. using the (q t À q f )g term in Eq. (1). In , gravitational sag in four clinical cases was compensated for using this term and encouraging results were presented. The data reported in used the same mechanism for simulating sag but only represent surface measurements. In the work presented here, surface and subsurface beads were tracked in a phantom under controlled gravitational loading conditions. As stated in Section 4.2, the computational model recaptured approximately 88.7% of the surface and subsurface shift. The second level of significance for the phantom experiments is in validating the proposed constrained linear inverse model approach for distributed surgical loads. Table 2 indicates an approximate 92% compensation capability when only using sparse surface data to guide the inverse model. One interesting aspect to observe in the phantom results is that the constrained linear inverse model outperformed the forward-based computational model that used the known boundary conditions. Undoubtedly the inaccuracies in the forward model are from inappropriate small-strain approximations, nonlinear material effects. Despite these inaccuracies and the lack of explicit drainage/incline information, the constrained linear inverse model delivered a modest improvement over the open-loop predictive model synthesizing a better match through the combination of the pre-computed basis solutions.. For the clinical studies, it was interesting to note that in both the patient cases, the atlas with mannitol-induced deformations recaptured most of the measured shift. Atlas III, mannitol-induced deformations with non-resected tumor recaptured most of the shift for Patient 1, while for Patient 2, Atlas IV, mannitol-induced deformations with resected tumor recaptured most of the measured shift. While no statistical significance can be inferred, it is interesting that mannitol was administered in both patients and that when comparing the results among the atlas', the predictions by mannitol induced shift are better than the gravity-induced shift. Although anecdotal, this may suggest that mannitol-induced shift may have a more prominent role in compensation strategies than previously reported. While these thoughts are intriguing, unfortunately, more detailed validation with subsurface measurements in a bigger patient population will be required to assert any conclusions. Nevertheless, the results among the experiments are markedly consistent and indicate that the constrained linear inverse modeling approach is a viable method for the compensation of distributed loading conditions.. The simulation results concerned with brain swelling were of comparable accuracy to the phantom and clinical experiments. In addition, the reported swelling shift magnitudes were comparable to those found in the literature (Dorward et al., 1998; Nimsky et al., 2000; Sun et al., 2005) . One common criticism of the MUIGNS systems is that tissue swelling cannot be accounted for. Initial results of the sensitivity of the inverse model to noise (as shown in Fig. 12 ) shows that the model is relatively insensitive to noise, as long as the displacement data are still contained by the atlas. The results shown here suggest that swelling conditions encountered in the OR can be simulated using computer models. Fig. 13c shows the distribution of the shift error recaptured in the tumor region. Though the error increased when compared to the error distributions on the surface containing the measurement nodes (shown in Fig. 13b ), the constrained linear inverse model still recaptured 83.6% of the mean shift in the vicinity of the tumor. These results combined with the target/sub-surface validations from the phantom experiments suggest that the constrained linear inverse model is a good framework for predicting sub-surface displacements using sparse intraoperative measurements.. While the work presented here is encouraging, the following issues need to be addressed before implementing this approach in a MUIGNS system: (i) more detailed validations with intraoperative imaging modalities such that the accuracy of the technique in predicting full volume displacements can be achieved; though validating the accuracy of the model has been reserved for a future study, the phantom results shown here and the simulation study results reported in Dumpuri et al. (2003) suggest that the model will behave in a similar fashion when predicting full volume displacement fields from sparse intraoperative data; (ii) sensitivity analysis of the inverse model to the particular selection of the boundary conditions and the consistency of the atlas; (iii) more detailed understanding of the internal structures affecting brain shift, e.g. the falx cerebri has been shown to inhibit cross-hemisphere movement; (iv) new studies focused on the improvement from subsurface data such as from coregistered ultrasound; and, (v) more studies regarding the sensitivity of the methods to the number and spatial distribution of sparse intraoperative data points. With respect to this last point, the results presented here have yielded a potentially important finding. In both the phantom and clinical experiments, the constrained linear inverse model was guided with a relatively modest number of points (12-15 points) spatially distributed on the area of observation. The level of model-fit in these cases is remarkable and makes it evident that assumptions regarding the extent, i.e. amount of data necessary for modelupdating can and should be challenged as these new systems are developed.. Previous work has demonstrated that modeling can predict deformations induced by surface loading conditions such as tissue retraction (Platenik et al., 2002; . Although detailed clinical studies have not been presented, the results suggest that the inverse model has the capability to predict intraoperative brain shift resulting from distributed loading conditions. These preliminary results indicate that the inverse model when combined with the approach reported in Platenik et al. (2002) and has the ability to predict introperative brain shift resulting from surface loads and distributed loads, thereby completing the MUIGNS framework.

O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O


FPGA implementation of a wavelet neural network with particle swarm optimization learning This paper introduces implementation of a wavelet neural network (WNN) with learning ability on field programmable gate array (FPGA). A learning algorithm using gradient descent method is not easy to implement in an electronic circuit and has local minimum. A more suitable method is the particle swarm optimization (PSO) that is a population-based optimization algorithm. The PSO is similar to the GA, but it has no evolution operators such as crossover and mutation. In the approximation of a nonlinear activation function, we use a Taylor series and a look-up table (LUT) to achieve a more accurate approximation. The results of the two experiments demonstrate the successful hardware implementation of the wavelet neural networks with the PSO algorithm using FPGA. From the results of the experiment, it can be seen that the performance of the PSO is better than that of the simultaneous perturbation algorithm at sufficient particle sizes. In recent years, artificial neural networks (ANN) have been widely used in many fields, such as chemistry [1] , diagnosis [2] , control [3] , and image processing [4] . The most commonly used artificial neural network is the multilayer perceptron (MLP) proposed by Gibson et al. [5] .The MLP is a full connection structure that uses sigmoid functions as hidden node functions. MLP equipped with global transfer functions active over a large range of input values has been widely used to perform global approximations [7] .. A suitable approach to overcome the disadvantages of global approximation networks is the substitution of the global activation function with localized basis functions. In this type of local network, only a small subset of the network parameters is engaged at each point in the input space. The network adaptation at a region of the input space does not corrupt the network parameters in more distant regions. Wavelet neural networks (WNN) are examples of spatially localized networks that have been applied to many fields. In this paper, implementation of wavelet neural networks with learning ability using FPGA was proposed. PSO is used to find an optimum solution and only some simple operations are required to adjust parameters. The results of two experiments demonstrate the successful hardware implementation of the wavelet neural networks with the PSO algorithm using FPGA. Some of the features of the wavelet neural networks with the PSO algorithm can be summarized as follows: (1) an analog wavelet neural network is realized based on digital circuits; (2) hardware implementation of the PSO learning rule is relatively easy; and (3) hardware implementation can take advantage of parallelism. From the results of the experiment with the prediction problem, it can be seen that the performance of the PSO is better than that of the simultaneous perturbation algorithm at sufficient particle sizes.. A major drawback of the existing wavelet neural networks is that their application domain is limited to static problems due to their inherent feedforward network structure. In the future, a recurrent wavelet neural network will be proposed for solving identification and control problems.

O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I I O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O


Design principles and user interfaces of Erkki Kurenniemi's electronic musical instruments of the 1960's and 1970's This paper presents a line of historic electronic musical instruments designed by Erkki Kurenniemi in the 1960's and 1970's. Kurenniemi's instruments were influenced by digital logic and an experimental attitude towards user interface design. The paper presents an overview of Kurenniemi's instruments and a detailed description of selected devices. Emphasis is put on user interface issues such as unconventional interactive real-time control and programming methods. Erkki Kurenniemi (b. 1941 ) is one of the leading pioneers of Finnish electroacoustic music. A musical instrument designer, composer, and multimedia artist, Mr. Kurenniemi played a central role in the Finnish avant-garde music scene in the 1960's and early 1970's. During that time, he built a series of unique electronic musical instruments, each of which can be regarded as unconventional also from today's perspective. He also founded the first and still operational Finnish electronic music studio at the Department of Musicology, University of Helsinki. Kurenniemi collaborated with several composers and artists including Ralph Lundsten, Erkki Salmenhaara, Osmo Lindeman, Henrik Otto Donner, Jukka Ruohomäki, M.A. Numminen, and the Norwegian theater company Scene 7.. Kurenniemi's best-known achievement as an instrument maker is a line of electronic music devices named Dimi (Digital Musical Instrument). As an instrument designer, KurenPermission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. niemi's particular interests were in digital logic and its application to user interface design. In the mid 1970's, Kurenniemi left the electroacoustic music scene to work for the Finnish industry specializing in industrial automation and robotics. Later he worked as a researcher in Heureka, The Finnish Science Centre. Kurenniemi became a well-known specialist and visionary of modern technology and appeared in many television and newspaper interviews describing his futuristic ideas and innovations. In the 1980's Kurenniemi returned to music-related research by developing a new theory of tonal harmony [5] .. The early 2000's saw a renewed interest in Kurenniemi The evolution of Kurenniemi's instrument designs may be regarded as a logical and gradual transition from analog to "quasi-digital" to digital electronics. In his electronics designs, Kurenniemi was more interested in programmability than in sonic flexibility or in developing novel synthesis techniques. In particular, Kurenniemi did not implement a purely digital sound synthesis system in any of his early or Dimi instruments. However, his user interface designs were greatly influenced by digital logic. This, combined by the lack of conventional control hardware, such as a piano-style keyboard (Dimi-O being the only exception), made adoption of these instruments difficult for ordinary musicians.. Despite their unconventional user interfaces, there is something appealing and fascinating in many of the instruments. In particular, the compact size and the austere but visually balanced user interface of Dimi-A, have made it a classic and gained it status as a work of art. The inter-artistic optical interface of Dimi-O and the somewhat provocative skin resistance interface of Dimi-S alone earned Kurenniemi a reputation of an eccentric artist and scientist. In some aspects, Dimi-S and Dimi-O remind us of the electronic instruments of Lev Sergeyevich Termen, a.k.a. Léon Theremin. On the other hand, the microprocessor based Dimi-6000, Kurenniemi's most flexible and technically advanced instrument, appeared only as a conventional CRT terminal and a set of stacked aluminum boxes.. Kurenniemi's visions often went beyond what was possible to implement with contemporary technology or what was financially realistic at the time. He was, nevertheless, successful as an instrument designer having completed a total of 9 different musical devices between 1968 and 1973. Since the marketing attempts for the Dimi instruments failed, and in spite of Kurenniemi's many appearances in the Finnish mass media, his instruments remained fairly unknown until the late 1990's when the Finnish electronic music scene adopted him as their forerunner and "godfather". As a result, Kurenniemi was celebrated as the guest of honor and featured artist of the 2002 Avanto international media art festival in Helsinki. There, many of his instruments were played live, perhaps for the largest audience ever.. Most of Kurenniemi's instruments have survived to present day and many of them are still in working order. The remaining instruments in Finland are stored, and occasionally used, in the Electronic Music Studio at the Department of Musicology, University of Helsinki.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Personally tailored teaching in WHURLE using conditional transclusion The emergence of Technology Based Learning has generated a number of pedagogic problems related to learner diversity.. In this paper we present an interim snapshot of a prototype XSLT / XML hypermedia learning environment able to respond adaptively to individual learner profiles using conditional transclusion. Technology Based Learning (TBL) -usually in the form of interactive hypermedia -is often represented in higher education as a panacea, a cost efficient means of maintaining teaching quality in the face of declining per capita resources. The rise of the World Wide Web (WWW) has added the dimension of geographical distance to this, allowing for the exciting possibility of globally distributed education [1, 2] . However, there remain a number of problems that make the implementation of effective TBL (especially on the WWW) a difficult task. Some of these problems are pragmatic, and others are pedagogic.. Although independent learning technologies are extremely good at delivering both information resources and declarative exposition, such forms of delivery constitute only one component of the independent learning process. Laurillard [3] has suggested a "conversational" model of interaction between students and teachers. At the heart of this model is an iterative process of dialogue in which both student and teacher describe their conceptions of the focal issue or topic, and the teacher responds to the student's input "adaptively" in accordance with perceived need. Although this adaptive model can, in principle, be implemented in software, in practice such sophistication is rare. One of the principal pedagogic weaknesses of existing TBL environments is 'static content' -i.e. the system subjects all students to much the same pedagogical experience, regardless of their personal abilities and learning requirements. In this short paper we will outline one of the key technical aspects in a prototype XSLT (eXtensible Stylesheet Language Transformations) implementation of a framework based upon XML (eXtensible Markup Language). This prototype is called WHURLE (Webbased Hierarchical Universal Reactive Learning Environment) [4] which combines the key requirements specified above in a single, Integrated Learning Environment (ILE). Such an environment models the teacher-learner dialogue, is adaptive to learner needs [5] , and processes content structurally predisposed to massproduction, reusability and global dissemination. We have proposed a method to use core XML technologies to implement a transclusive mechanism. This allows the delivery of material suited to each learner, and is envisaged to be a key step in making TBL more efficient and valuable to both the learner and the teacher.

O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O B I O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


The trustworthiness analyzing of interacting business process based on the induction information Under the open environments, it is very difficult to guarantee the trustworthiness of interacting business process using traditional software engineering methods, at the same time, for dealing with the influence of external factors, some proposed business process mining methods are only effective 1-bounded business process, and some behavior dependent relationships are ignore. A behavior trustworthiness analysis method of business process based on induction information is presented in the paper. Firstly, aimed to the internal factors, we analyze the consistent behavior relativity to guarantee the predictable function. Then, for the external factors, in order to analyze the behavior change of business process, we propose a process mining methods based on induction information. Finally, experiment simulation is given out, and compares our method with genetic process mining methods. Theoretical analysis and experimental results indicate that our method is better than the genetic process mining method. With the tremendous growth of information and communication technologies, it is advantageous to design and implement the complex inter-enterprise business processes. One of the major innovations is the concept of serviceoriented architectures (SOA) which considers software systems as being made up with autonomous, dynamic, loosely coupled and service-based components. For interacting business processes, especially those that run over the Internet, behavior trustworthiness plays a crucial role. The operating and development environment of business process turn from the traditional closed static environment and extend to an open, dynamic, ever-changing network environment [1] . In the network computing environment, the behavior of the entity is uncontrollable and uncertain, which posed a severe challenge for behavior trustworthiness of the running processes [2] . Therefore, the guarantee of the behavior trustworthiness of the business under the network environment has become the industry's research focus.. Analysts often see lack of trustworthy methods as a major impediment to the adoption of web services in building agile business process. Software behavior is a running operation series, which can describe the software some feature when interacting with others. By analyzing the definition of trustworthiness, that the behavior can be predictable is a key property. So, achieving behavior trustworthy goals often relies on an appropriate use of software behavior. However, the notion of behavior trustworthiness is often neglected in business process, which usually concentrates on modeling the process in a way that functional correctness and security can be shown, either manually or using formal proof tools like model checking and some static validation methods. In contrast to features that are crucial for functional correctness and security features, and so on, behavior trustworthiness is typically integrated in an application, which mainly assures the execution behavior and computing result of the business process to be consistent with the expected effects. The integration of trustworthy features into a business process is not well understood. Especially, the trustworthy features are difficult to describe, and to analyze quantitatively [3] . So, the problem need further investigation, and seek new methods to deal with the behavior trustworthiness.. The interacting business process may run into two kinds of problems, one problem is the interactive influence between the business processes in the building process; one component business process may impact the other, which leads to the function of composite business process to be improper. The other problem is that the composite business process may be influenced by the outer factors. Aimed to the internal interactive influence, the interaction behavior relativity of business process need be studied. For the outer factors, we use the business process mining methods to construct the actual behavior model based on the running logs. At present, some process mining methods [4] [5] [6] have been proposed, but the existing methods are deficient to solve the problem. In the paper, our contributions are given out the analyzing methods of the behavior trustworthiness about business process based on the induction information; the methods can deal with kbounded business process, and take the indirect dependent relationships into account.. The rest of this paper is organized as follows. Section 2 introduces related work. In Section 3 discusses motivating examples and research framework. Section 4 details propose the analyzing of the composite model of business process based on Petri net. Section 5 presents Building the K-bounded process behavior model based on new process mining methods. Simulation experiments are given out in the Section 6. Finally, Conclusions and acknowledgements are in Section7 and Section 8 respectively. At present, the trustworthiness of business process is a study focus, the essence characteristics of trustworthy business process are the execution result and behavior can be predictable. Under the open and dynamic changing environment, some complex demands need several business processes interacting to implement, these cause the composite business process to behave uncontrollably, uncertainly, and unpredictably. The paper presents a behavior trustworthiness analysis method of business process based on induction information. Firstly, aimed to the internal factors, we analyze the consistent behavior relativity in order to guarantee the predictable function of business process. Then, for the external factors, in order to analyze the behavior change of business process, we propose a process mining methods based on induction information. Finally, using methods above mentioned, we test our methods in large Benchmark, and compare our methods with genetic process mining. Theoretical analysis and experimental results indicate that our method is better than the methods of genetic process mining.. Based on the theoretical analysis and experimental results, the innovation and advantage of the paper are: 1) In order to analyze the predictable interaction function, consistent behavior relativity based on Petri net is presented. 2) An effective measure methods of behavior conformance is proposed, the methods can solve these problems such an Fig.14 and Fig.15 , but these problems can not solved in literature in [20] . 3) Business process mining methods based on induction information is proposed, which can take into account induction information from the theoretical composite model, and avoid the blindness of building process model.. In the future, we would like to study the trustworthy evaluation of networked software, and study the adaptation methods of non-consistent behavior relativity. Moreover, it is also one of our future works to study the intelligence and dynamic behavior analyzing methods of complex business process. The study of business process mainly takes advantage of some methods that described the interaction between different components of the system. Research on business process modeling has recently started to encode business-process diagrams into a formal model that can be given a suitable semantics, usually based on interacting state machines [7] , Petri nets [8], or graph grammars [9] . Models that explicitly address the incorporation of security issues in the design process are typically extensions of a fragment of UML that can be given the desired semantics. They address more general notions of security than in the standards like multi-level security [10] or rolebased access control [11] . A concrete counterexample is given in [12] . Being more precise, there is no guarantee that properties of these abstractions are also valid for the concrete implementation. In the literature [13] , the paper presents a general methodology for integrating arbitrary trusted requirements in the development of business processes in a both elegant and rigorous way, and show how trusted relationships between different parties and their respective security goals can be reflected in a specification, which results in a realistic modeling of business processes in the presence of malicious adversaries. Special attention is given to the incorporation of cryptography in the development process with the main goal of achieving specifications that are sufficiently simple to be suited for formal verification, yet allow for a provably secure cryptographic implementation. [14] described a business process driven framework (called the BPD-ACS) for developing both the model and formulating the access decision rules. The model used is the Role Based Access Control (RBAC) model and the access decision rules are based on temporal business associations. In the literature [15] , analysis of workflow dynamic changes based on Petri net is proposed, but did not take the behavior relativity into account. [16] proposed the logic-based approach about specification and verification of declarative open interaction of business process, in order to guarantee the trustworthiness and reliability of the developed model, as well as to ensure that the actual executions of the system effectively comply with it. [17] [18] proposed a business process to be trustworthy if the behavior of all services is trustworthy, and gave out a verification mechanism through which the trustworthiness of a business process can be verified. Collaborative business processes often consist of services provided by multiple business entities which agree to join business collaboration. To enable trustworthy and secure consumption and provisioning of services across organizational boundaries, security requirements must be carefully defined so as to be coherent, consistent, and in compliance with designed business processes. However, managing security requirements in collaborative environments is error-prone, effort inefficient, and hard to be verified. [19] discussed the architecture specific to this issue, as an add-on to a trustworthy service-oriented architecture, and proposed suitable formal notations and formal analysis in the construction of this automated facility.. All works mentioned above did not study the behavior trustworthiness of business process, only the function correctness or security, so it is necessary to study new methods to analyze the behavior trustworthiness of interactive business processes.

O B O O O B I O O O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O B O O O B I O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B O O O B I O O O O O O O O O O O O O B I I O O O O O O O O B I O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O O O O O O O O O O O O B O O O O


Interface

grammars

for

modular

software

model

checking

We

propose

an

interface

specification

language

based

on

grammars

for

modular

software

model

checking.

In

our

interface

specification

language,

component

interfaces

are

specified

as

context

free

grammars.

An

interface

grammar

for

a

component

specifies

the

sequences

of

method

invocations

that

are

allowed

by

that

component.

Using

interface

grammars

one

can

specify

nested

call

sequences

that

cannot

be

specified

using

interface

specification

formalisms

that

rely

on

finite

state

machines.

Moreover,

our

interface

grammars

allow

specification

of

semantic

predicates

and

actions,

which

are

Java

code

segments

that

can

be

used

to

express

additional

interface

constraints.

We

have

built

an

interface

compiler

that

takes

the

interface

grammar

for

a

component

as

input

and

generates

a

stub

for

that

component.

The

resulting

stub

is

a

table-driven

parser

generated

from

the

input

interface

grammar.

Invocation

of

a

method

within

the

component

becomes

the

lookahead

symbol

for

the

stub/parser.

The

stub/parser

uses

a

parser

stack,

the

lookahead,

and

a

parse

table

to

guide

the

parsing.

The

semantic

predicates

and

semantic

actions

that

appear

in

the

right

hand

sides

of

the

production

rules

are

executed

when

they

appear

at

the

top

of

the

stack.

We

conducted

a

case

study

by

writing

an

interface

grammar

for

the

Enterprise

JavaBeans

(EJB)

persistence

interface.

Using

our

interface

compiler

we

automatically

generated

an

EJB

stub

using

the

EJB

interface

grammar.

We

used

the

JPF

model

checker

to

check

EJB

clients

using

this

automatically

generated

EJB

stub.

Our

results

show

that

EJB

clients

can

be

verified

efficiently

using

our

approach.

The

application

of

model

checking

techniques

directly

to

programs

has

shown

promise

for

specific

verification

tasks,

such

as

checking

for

concurrency

errors

or

checking

device

drivers

for

interface

violations

However,

there

are

two

related

problems

that

hinder

applicability

of

model

checking

to

software

in

a

wider

scale:

state

space

explosion

(i.e.,

exponential

increase

in

the

search

space

by

increasing

number

of

variables

and

concurrent

components)

limits

the

scalability

of

model

checking

techniques

and

environment

generation

(i.e.,

finding

models

for

parts

of

software

that

are

outside

the

scope

of

the

model

checker)

limits

the

applicability

of

model

checking

to

the

domains

where

such

environment

models

are

available..

Our

focus

in

this

paper

is

model

checking

Java

programs.

The

limitations

we

mentioned

above

are

apparent

in

Java

Path

Finder

(JPF)

which

is

a

model

checker

for

Java

programs.

JPF

cannot

handle

native

calls

in

Java

programs.

Hence,

in

order

to

use

JPF

for

verification

of

Java

programs,

one

has

to

write

environment

models

for

any

component

that

uses

native

code,

which

is

a

daunting

task..

Note

that

inability

to

handle

native

code

is

not

only

a

limitation

specific

to

JPF,

but

it

is

the

sign

of

an

inherent

problem

in

model

checking.

In

order

to

search

the

state

space

of

a

program

exhaustively

(as

most

model

checkers

attempt

to

do),

one

needs

a

representation

of

that

state

space.

JPF

chooses

to

model

the

state

space

of

a

Java

program

by

recording

configurations

of

the

Java

Virtual

Machine

(JVM).

JPF

has

its

own

JVM

which

keeps

track

of

different

configurations

that

are

visited

during

the

execution

of

the

program

that

is

being

verified.

Execution

of

native

code,

by

definition,

moves

the

program

execution

outside

the

scope

of

the

JVM

and

hence

cannot

be

observed

by

JPF.

Even

if

one

tries

to

keep

track

of

program

execution

at

a

lower

level

of

abstraction,

perhaps

by

keeping

track

of

the

physical

memory

and

processor

state,

a

similar

problem

will

arise

if

one

tries

to

analyze

a

distributed

program

which

involves

interactions

among

multiple

machines

or

a

program

that

interacts

with

a

database

server,

etc.

Eventually,

this

will

require

keeping

track

of

the

state

of

each

and

every

component

that

the

program

interacts

with.

This

is

unlikely

to

be

a

scalable

approach

due

to

the

state

space

explosion.

Moreover,

in

many

Figure

An

overview

of

our

approach

(if

not

the

majority)

of

cases,

the

developer

who

is

trying

to

check

the

correctness

of

a

program

may

not

have

access

to

the

code

of

all

the

components

that

the

program

interacts

with..

In

this

paper,

we

propose

a

semi-automated

approach

to

attack

the

above

mentioned

problems.

We

propose

an

interface

specification

language

and

require

the

users

to

write

interface

specifications

for

components

that

are

outside

the

scope

of

the

current

verification

effort.

Our

interface

specification

language

allows

a

user

to

write

an

interface

grammar

for

a

component

to

specify

the

constraints

on

the

ordering

of

calls

made

by

the

program

to

that

component.

This

approach

enables

modeling

of

nested

call

structures

that

cannot

be

expressed

by

interfaces

based

on

finite

state

machines.

Moreover,

in

order

to

provide

a

flexible

approach

that

can

handle

complex

interface

constraints,

our

interface

specification

language

allows

the

users

to

escape

to

Java

and

write

semantic

predicates

or

actions

in

Java,

specifying

the

behavior

of

the

component

(similar

to

the

approach

used

by

parser

generators

such

as

Yacc

We

believe

that

our

approach

provides

a

balance

between

two

extreme

alternatives,

i.e.,

writing

stubs

completely

manually

or

automatically

extracting

simple

abstract

models

such

as

finite

state

machines.

Figure

shows

an

overview

of

our

approach.

We

have

built

an

interface

compiler

that

takes

an

interface

grammar

as

input

and

automatically

generates

a

stub

for

the

corresponding

component.

The

component

stub

is

a

tabledriven

top-down

parser

that

parses

the

sequence

of

incoming

method

calls

(i.e.,

the

method

invocations)

based

on

the

grammar

provided

in

the

interface

specification.

During

execution,

the

stub/parser

executes

the

semantic

predicates

and

actions

at

the

appropriate

times

based

on

their

placement

in

the

productions

of

the

interface

grammar.

If

the

program

that

uses

the

component

violates

the

component's

interface,

then

the

component

stub

either

reports

a

parse

error

(that

corresponds

to

an

error

in

the

call

sequence)

or

a

semantic

predicate

violation

(which

can

correspond

to

an

error

in

an

argument

that

is

passed

to

the

component)..

In

order

to

write

compact

interfaces

it

is

necessary

to

support

nondeterminism

in

an

interface

specification

language.

A

stub

generated

from

an

interface

specification

of

a

component

should

generate

an

over-approximation

of

the

behavior

of

that

component.

The

use

of

nondeterminism

allows

specification

of

a

set

of

behaviors

in

a

concise

manner.

Our

interface

specification

language

provides

a

nondeterministic

switch

operator,

i.e.,

two

or

more

switch

cases

evaluate

to

true,

then

one

of

them

is

selected

nondeterministically..

We

assume

that

the

target

software

model

checker

provides

a

nondeterministic

choice

primitive.

Our

current

interface

compiler

uses

the

nondeterministic

choice

primitives

provided

by

JPF.

We

can

easily

modify

our

interface

compiler

to

support

other

model

checkers

as

long

they

support

nondeterministic

choice.

During

verification,

the

model

checker

exhaustively

checks

all

possible

choices

that

can

result

from

the

use

of

nondeterministic

choice

primitives.

While

generating

the

stub

code,

our

interface

compiler

converts

the

nondeterministic

choices

in

the

interface

grammar

to

calls

to

the

nondeterministic

choice

primitive

of

the

model

checker.

This

means

that

all

possible

behaviors

provided

by

the

interface

will

be

checked

by

the

model

checker

during

verification

of

the

program

with

the

automatically

generated

stub..

Our

approach

enables

model

checking

to

be

executed

in

a

modular

fashion

by

replacing

different

components

in

the

software

system

with

environment

models

generated

from

their

interfaces.

We

conducted

a

case

study

to

demonstrate

our

approach

using

Enterprise

JavaBeans

(EJB)

Persistence

API

clients.

We

wrote

an

interface

grammar

for

the

Persistence

API

and

verified

Persistence

API

clients

using

a

stub

automatically

generated

from

this

interface.

Our

experimental

results

demonstrate

that

interface

grammars

can

be

used

effectively

in

modular

verification..

The

rest

of

the

paper

is

organized

as

follows.

Section

provides

an

overview

of

our

interface

specification

language.

Section

discusses

the

interface

compiler.

Section

discusses

the

EJB

case

study.

Section

includes

a

discussion

on

related

work,

and

Section

concludes

the

paper.

We

have

proposed

and

implemented

a

new

framework

for

conducting

modular

software

model

checking

based

on

interface

grammars.

We

proposed

an

interface

specification

language

based

on

interface

grammars

and

we

built

a

compiler

that

automatically

generates

stubs

for

components

using

interface

specifications

written

in

our

interface

specification

language.

We

have

used

this

tool

to

conduct

model

checking

relating

to

the

key

interfaces

of

the

Enterprise

JavaBeans

Persistence

API,

and

have

demonstrated

that

our

approach

is

feasible

and

efficient.

In

future

work,

we

would

like

to

apply

our

interface

grammars

to

model

checking

of

concurrent

programs,

as

well

as

the

generation

of

object

graphs

for

model

checking.

Use

of

finite

state

machines

for

specification,

verification

and

extraction

of

interfaces

have

been

studied

extensively

Finite

state

machines

cannot

specify

nestedcall

structures

such

as

the

recursive

transaction

example

we

use

in

this

paper.

The

interface

grammars

we

propose

in

this

paper

enables

us

to

specify

such

interactions.

Moreover,

we

believe

that

the

semantic

predicates

and

actions

that

are

allowed

in

our

interface

grammars

are

necessary

to

model

interfaces

of

complex

components.

Another

factor

that

differentiates

our

work

from

that

of

is

that

we

do

not

extract

interfaces,

rather,

we

use

interface

grammar

specifications

to

check

both

interface

conformance

and

also

to

achieve

modular

verification..

The

Specification

Language

for

Interface

Checking

(SLIC)

is

used

to

specify

interface

constraints

in

the

SLAM

project

In

SLIC,

interfaces

are

specified

using

state

machines.

The

transitions

of

state

machines

are

associated

with

C

statements

that

can

be

used

to

specify

additional

constraints

on

the

interface.

As

with

the

other

state

machine

based

approaches

discussed

above,

the

approach

used

by

SLIC

is

not

appropriate

for

specification

of

nested

call

sequences..

In

finite

state

interface

specifications

are

used

to

achieve

modular

verification

where

behavior

verification

and

interface

verification

are

executed

as

two

separate

steps.

Interface

grammars

proposed

here

provide

a

richer

language

for

specification

of

interfaces

and

can

be

integrated

to

the

modular

verification

approach

used

in

Environment

generation

is

a

critical

problem

for

achieving

modularity

in

software

model

checking

and

has

been

studied

before.

presents

techniques

for

automatically

closing

environments

of

open

reactive

programs

by

automatically

creating

the

most

general

environment

for

the

program

using

dataflow

analysis.

on

the

other

hand,

investigates

automatically

generating

environments

for

components

using

side

effect

and

points-to

analyses

for

modular

model

checking.

We

use

a

semi-automated

approach

where

the

user

writes

an

interface

grammar

and

the

interface

grammar

is

automatically

compiled

to

a

component

stub

for

modular

verification.

We

believe

that

for

specification

of

rich

interfaces

such

as

the

EJB

interface

discussed

in

this

paper

it

is

necessary

to

get

user

input

in

order

to

restrict

the

behaviors

allowed

by

the

interface..

The

Bandera

environment

generator

discussed

in

also

uses

a

semi-automated

approach

in

which

environment

models

are

automatically

synthesized

from

environment

assumptions.

The

environment

assumptions

are

given

as

LTL

formulas

or

regular

expressions

specifying

ordering

of

program

actions

which

are

unit

method

calls

or

field

assignments

that

can

be

executed

by

the

environment.

Our

approach

based

on

interface

grammars

enables

us

to

specify

nested

call

sequences

that

cannot

be

expressed

using

formalisms,

such

as

LTL

or

regular

expressions,

that

can

be

recognized

by

finite

state

machines.

Also

rather

than

focusing

on

environment

generation,

we

are

focusing

on

specification

of

interfaces.

Of

course,

these

are

closely

related

concepts

since

the

interfaces

of

components

that

interact

with

a

program

forms

the

environment

of

that

program.

However,

we

believe

that

it

is

more

likely

for

developers

to

write

interface

specifications

for

different

components

rather

than

writing

an

environment

for

a

particular

program.

We

have

applied

these

grammars

to

several

test

cases

from

the

Hibernate

implementation.

In

some

sense

these

are

excellent

measures

of

the

fidelity

of

our

interface;

since

they

were

written

to

expose

errors

in

Hibernate

they

should

similarly

expose

errors

in

our

simulation

of

the

Persistence

API.

As

well,

the

test

cases

include

some

invalid

clients

that

trigger

exceptions;

we

can

use

these

to

verify

clients

against

the

interface,

marking

clients

with

erroneous

behavior..

We

analyze

the

following

test

cases:.

EntityManagerTest.testContains

tests

minimal

normal

functionality,

like

persisting

an

object

and

retrieving

it

under

its

primary

key.

It

also

ensures

that

trying

to

check

the

status

of

a

non-manageable

object

will

fail

with

an

exception..

EntityManagerTest.testClear

ensures

that

objects

managed

by

the

EntityManager

transition

to

the

detached

state

after

a

clear..

EntityManagerTest.testPersistNoneGenerator

ensures

that

a

simple

object

is

equal

to

itself

after

it

has

been

persisted

and

reloaded..

EntityManagerTest.testIsOpen

verifies

that

an

EntityManager

is

open

upon

creation

and

stays

that

way

until

it

is

closed..

AssociationTest.testBidirOneToOne

verifies

that

persisting

one

half

of

a

bidirectional

association

will

persist

the

other

half

as

well..

AssociationTest.testMergeAndBidirOneToOne

verifies

that

the

bidirectional

association

works

even

with

detached

objects..

CallbacksTest.testCallBackListenersHeirarchy

verifies

that

methods

tagged

with

@PrePersist

are

called

when

the

object

in

question

is

persisted..

CallbacksTest.testException

verifies

that

methods

in

other

classes

that

have

a

declared

@EntityListeners

relationship

with

the

persisted

object

are

also

called.

The

name

comes

from

the

method

that

is

to

be

called,

which

throws

an

exception..

GetReferenceTest.testWrongIdType

verifies

that

asking

for

objects

using

the

wrong

primary

key

type

is

an

illegal

operation..

ExceptionTest.testEntityNotFoundException

verifies

that

nonexistent

objects

fetched

with

getReference

should

raise

exceptions

when

they

are

referred

to..

InheritanceTest.testFind

verifies

that

if

A

is

a

subclass

of

B,

persisting

an

instance

of

A

and

asking

for

all

Bs

should

retrieve

the

first

object..

testAlwaysTransactionalOperations

method

of

FlushAndTransactionTest

class

checks

that

flushes

and

locks

are

only

valid

from

within

transactions..

We

conducted

our

experiments

using

a

Linux

machine

with

GHz

Pentium

with

gigabytes

of

memory.

We

ran

each

of

the

test

cases

listed

above

in

two

configurations

on

our

testbed.

We

first

ran

the

test

case

normally,

to

test

the

validity

of

our

interface.

We

achieved

a

success

rate

in

this,

showing

that

our

interface

successfully

simulates

the

behavior

of

EJB

Persistence

API.

Then

we

ran

it

again

in

a

configuration

where

any

exceptions

seen

in

the

parsing

trigger

fatal

runtime

errors;

these

test

whether

the

clients

are

faithful

to

the

Persistence

API.

Of

our

test

cases,

some

of

the

deliberately

perform

illegal

operations,

which

are

caught.

Our

results

are

shown

in

Table

We

were

unable

to

measure

run

times

in

increments

of

less

than

a

second,

as

JPF

itself

does

not

support

getting

the

current

time

from

within

the

running

program,

and

provides

only

second

and

megabyte

resolution

for

its

output.

Even

with

this

limited

resolution,

our

results

demonstrate

that

using

our

approach

verification

of

both

nontrivial

client

code

and

the

interface

specification

can

be

done

efficiently

using

relatively

little

memory

and

execution

time.

B I O B B I I O O O O B I O O O O B B I I O O O B I B B O O O B O O O O B O O B O O O O B O O O O O O B O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O B O B O O O O O O B O O O O O O B O O O B O B O O O B O O B O O O O O O O O O O O B O O O O O O O O O O B O B O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O B O O B O O B O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O B O B O O O O O B O O O B O O B O O O B O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O B O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O B I O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O B O O B O O O O O O B O O B O O O B O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B I I O O B I O O O O B I O O B I O O O O O O O O O O O O O O O O O O B I O O O O O O O B O O O O O B O O B O B O O O O O O O O O O O O O O O O O O O O B I O B O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B O O O O O O O O B I O O O B B O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O B I O B I O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O B O O O O O O B I O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O B O O O B O O O O O B O O B I O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O


Semantic segmentation of images exploiting DCT based features and random forest This paper presents an approach for generating class-specific image segmentation. We introduce two novel features that use the quantized data of the Discrete Cosine Transform (DCT) in a Semantic Texton Forest based framework (STF), by combining together colour and texture information for semantic segmentation purpose. The combination of multiple features in a segmentation system is not a straightforward process. The proposed system is designed to exploit complementary features in a computationally efficient manner. Our DCT based features describe complex textures represented in the frequency domain and not just simple textures obtained using differences between intensity of pixels as in the classic STF approach. Differently than existing methods (e.g., filter bank, etc.) just a limited amount of resources is required. The proposed method has been tested on two popular databases: CamVid and MSRC-v2. Comparison with respect to recent state-of-the-art methods shows improvement in terms of semantic segmentation accuracy. This paper describes an approach for semantic segmentation of images. Two novel texture features based on DCT data are introduced in the Semantic Texton Forest framework [13] . The proposed DCT features describe complex textures capable to recognize object and region with different frequencies characteristics. Our approach makes use of a limited amount of resources that allow good accuracy for real time applications. The effectiveness of the proposed semantic segmentation system has been demonstrated by comparing it with the STF and other state-of-the-art approaches. In most of the case, our approach shows better performance overcoming the per-classes accuracy in the considered databases. Moreover, in a real scenario our system could show further improvements since usually a large version of the image is available in the pipeline. This avoids to perform the proposed upsampling block in the pipeline and generating a more reliable DCT data that are not affected by the interpolation. To address the challenges described above, different segmentation methods were proposed in literature.. Some basic approaches segment and classify each pixel in the image using a region-based methodology as in [14, 15, 16, 17, 18, 19, 20, 21, 22] . Other approaches use a multiscale scanning window detector such as Viola-Jones [23] or Dalal-Triggs [24] , possibly augmented with part detectors as in Felszenszwalb et al. [25] or Bourdev et al. [26] . More complex approaches as in [27, 28] unify these paradigms into a single recognition architecture, and leverage on their strengths by designing region-based specific object detectors and combining their outputs. By referring to the property that the final label of each pixel can 3 be dependent by the labels assigned to other pixels in the image, different methods use probabilistic models such as the Markov Random Field (MRF) and the Conditional Random Fields (CRF) that are suitable to address label dependencies. As example, the nonparametric model proposed in [29] requires no training and can easily scaled to datasets with tens of thousands of images and hundreds of labels. It works by scene-level matching with global image descriptors, followed by superpixel-level matching with local features and efficient MRF based optimization for incorporating neighbourhood context. In [30] , instead, a framework is presented for semantic scene parsing and object recognition based on dense depth maps.. Five view independent 3D features that vary with object class are extracted from dense depth maps at a superpixel level for training a randomized decision forest. which produce segmentations with precise object boundaries. In [34] a novel formulation is proposed for the scene-labelling problem capable to combine object detections with pixel-level information in the CRF framework. Since object detection and multi-class image labelling are mutually informative dependent problems, pixel-wise segmentation can benefit from the powerful object detectors and vice versa. The main contribution of [34] lies in the incorporation of top-down object segmentations as generalized robust potentials into the CRF formulation. These potentials present a principled manner to convey soft object segmentations into a unified energy minimization framework, enabling joint optimization and thus mutual benefit for both problems. A probabilistic framework is presented in [35] for reasoning about regions, objects, and their attributes such as object class, location, and spatial extent. The proposed CRF is defined on pixels, segments and objects. segmentation problem in terms of performance, they represent a bottleneck in the computation, because the inference is a highly resources consuming process. A powerful approach with good performance, while preserving high efficiency, is based on the random forest. For example in [13] , the authors show that one can build rich Texton codebooks without computing expensive filter-banks or descriptors, and without performing costly k-means clustering and nearest-neighbour assignments. Specifically, the authors propose the bag of Semantic Textons that is an extension of the Bag of Word Model [36] obtained by combining a histogram of the hierarchical visual word with a region prior category. In this paper we build on the semantic texton forest (STF) approach (detailed in Section 3) by proposing a new semantic segmentation pipeline which exploits features extracted on the DCT domain. The exploitation of contextual and structural information have been recently proposed in [37, 38] to improve the performances of random forests for semantic image labelling. In particular, the random forest approach has been augmented in order to consider topological distribution of object classes in a given image. A novel splitting function has been also introduced to allow the random forest to work with the structured label space. Recent trends also consider Convolutional Neural Network for the semantic segmentation. In [39] the authors adapt state-of-the-art classification networks (i.e., AlexNet, the VGG net, and GoogLeNet) into fully convolutional networks and transfer their learned representations to the segmentation task. A novel architecture that combines semantic information of the different layers is proposed to produce the final semantic segmentation of images.

B I O O O B O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O