Eigenvalue decay: A new method for neural network regularization This paper proposes two new training algorithms for multilayer perceptrons based on evolutionary computation, regularization, and transduction. Regularization is a commonly used technique for preventing the learning algorithm from overfitting the training data. In this context, this work introduces and analyzes a novel regularization scheme for neural networks (NNs) named eigenvalue decay, which aims at improving the classification margin. The introduction of eigenvalue decay led to the development of a new training method based on the same principles of SVM, and so named Support Vector NN (SVNN). Finally, by analogy with the transductive SVM (TSVM), it is proposed a transductive NN (TNN), by exploiting SVNN in order to address transductive learning. The effectiveness of the proposed algorithms is evaluated on seven benchmark datasets. One of the problems that occur during syntactic classifier training is called overfitting. The error on the training dataset is driven to a small value; however, the error is large when new data are presented to the trained classifier. It occurs because the classifier does not learn to generalize when new situations are presented. This phenomenon is related to the classifier complexity, which can be minimized by using regularization techniques [1] and [2] . In this paper we will apply regularization to improve the classification margin, which is an effective strategy to decrease the classifier complexity, in Vapnik sense, by exploiting the geometric structure in the feature space of the training examples.. There are three usual regularization techniques for neural networks (NNs): early stopping [3] , curvature-driven smoothing [4] , and weight decay [5] . In the early stopping criterion the labeled data are divided into training and validation datasets. After some number of iterations the NN begins to overfit the data and the error on the validation dataset begins to rise. When the validation error increases during a specified number of iterations, the algorithm stops the training section and applies the weights and biases at the minimum of the validation error to the NN. Curvature-driven smoothing includes smoothness requirements on the cost function of learning algorithms, which depend on the derivatives of the network mapping. Weight decay is implemented by including additional terms in the cost function of learning algorithms, which penalize overly high values of weights and biases, in order to control the classifier complexity, which forces the NN response to be smoother and less likely to overfit. This work introduces and analyzing a novel regularization scheme, named eigenvalue decay, aiming at improving the classification margin, as will be shown in Section 3.. In the context of some on-the-fly applications, the use of SVM with nonlinear kernels requires a prohibitive computational cost, since its decision function requires a summation of nonlinear functions that demands a large amount of time when the number of support vectors is big. Therefore, a maximal-margin neural network [6] [7] [8] [9] can be a suitable option for such kind of application, since it can offer a fast nonlinear classification with good generalization capacity. This work introduces a novel algorithm for maximum margin training that is based on regularization and evolutionary computing. Such method is exploited in order to introduce a transductive training method for NN.. The paper is organized as follows. In Section 2 we propose the eigenvalue decay, while the relationship between such regularization method and the classification margin is analyzed in Section 3. In Section 4 it is proposed a new maximum-margin training method based on genetic algorithms (GA) that is extended to the transductive approach in Section 5. Section 6 reports the experiments, while Section 7 summarizes some conclusions. The analysis presented in this paper indicates that by applying eigenvalue decay it is possible to increase the classification margin, which improves the generalization capability of NNs. The introduction of eigenvalue decay allowed the synthesis of two novel SVM-like training methods for NNs, including a transductive algorithm. These methods are suitable options for a faster nonlinear classification, by avoiding the time expensive decisionfunction of nonlinear SVMs, which may hinder on-the-fly applications, such as pedestrian detection (e.g. see Section 4.2 of [15] ). The experiments indicate that, regarding the classification accuracy, SVNN and TNN are similar to nonlinear SVM and TSVM; however, regarding the testing time, the proposed methods were significantly faster than nonlinear SVMs. The experiments also indicate that TNN can take advantage of unlabeled data, especially when few labeled data are available, as can be seen in Table 4 .. Oswaldo Ludwig received the M.Sc. degree in electrical engineering from Federal University of Bahia, Salvador, Brazil, in 2004 and the Ph.D. degree in electrical engineering from the University of Coimbra, Coimbra, Portugal, in 2012. He is an Assistant Professor of Computer and Electrical Engineering at the University of Coimbra. His current research focuses on machine learning with application on several fields, such as pedestrian detection in the domain of intelligent vehicles and biomedical data mining.. Urbano Nunes received the Lic. and Ph.D. degrees in electrical engineering from the University of Coimbra, Coimbra, Portugal, in 1983 and 1995, respectively. He is a Full Professor with the Computer and Electrical Engineering Department of Coimbra University. He is also the Vice Director of the Institute for Systems and Robotics where he is the Coordinator of the Automation and Mobile Robotics Group. He has been involved with/ responsible for several funded projects at both national and international levels in the areas of mobile robotics, intelligent vehicles, and intelligent transportation systems. His research interests are several areas in connection with intelligent vehicles and human-centered mobile robotics with more than 120 published papers in these areas.. Rui Araujo received the M.Sc. degree in Systems and Automation, in 1994 and the Ph.D. degree in electrical engineering from the University of Coimbra, in 2000, both from the University of Coimbra, Coimbra, Portugal. He is an Assistant Professor at the Department of Electrical and Computer Engineering of the University of Coimbra. His research interests include robotics, learning systems, control systems and theory, fuzzy systems, neural networks, real-time systems, architectures, and systems for controlling robots. In this section our methods are evaluated by means of experiments in three UCI benchmark datasets 4 and four datasets from [22] . 5 Table 1 details the applied datasets. 77 277 Haberman 3 153 153 306 Hepatitis 19 77 78 155 BCI 117 200 200 400 Digit1 241 750 750 1500 g241c 241 750 750 1500 Text 11 960 750 750 1500 The highly unbalanced datasets, Breast Cancer, Haberman, and Hepatitis, were introduced in our experimental analysis in order to verify the behavior of the optimization algorithms of transductive methods when working under the constraint on the predicted class proportion, i.e. the last term of (31). The other datasets are usually applied to evaluate semi-supervised learning algorithms. Each dataset was randomly divided into 10-folds, thus, all the experimental results were averaged over 10 runs.. The datasets were randomly divided into 10-folds, in order to average the results over 10 runs. Each fold contains all the data divided into two subsets: half for training and half for testing. For each training dataset, i.e. half of the data, it was performed 10folds cross-validation to set the classifier parameters. Therefore, the parameter setting does not take into account information from the testing dataset. In the case of SVM, the soft-margin parameter C and the RBF parameter γ (in the case of RBF kernel) were chosen in the set f10 À 3 ; 10 À 2 ; …; 10 3 g. In case of SVNN and TNN, the parameters C 1 , C 2 , and C 3 were chosen in the set f10 2 ; 10 3 ; 10 4 ; 10 5 g. Table 2 gives the averaged number of hidden neurons adopted after the cross-validation.. In our experiments SVNN and TNN were compared with NNs trained by the usual Levenberg-Maquardt (LM), as well as, SVM and TSVM. 6 Since SVNN and TNN can perform nonlinear classification, we also evaluated the performance of SVM and TSVM with RBF kernel. In order to verify the capability of transductive algorithms in learning from few labeled data, all the algorithms were trained by using only 10 labeled training points, as well as, all the training data. In both cases, the transductive algorithms made use of all the testing (unlabeled) data. Regarding the GA, the selective pressure was set to a¼ 6 and the population N pop ¼ 3000. Table 3 Accuracy (Acc), balanced error rate (BER), and positive predictive value (PPV) of inductive methods with 10 labeled training points. Tables 3-6 summarize the experimental results. Eqs. (32)-(34) define the three indexes adopted to assess the learning performance, i.e. accuracy (Acc), balanced error rate (BER), and a measure of precision named positive predictive value (PPV). In order to evaluate the influence of eigenvalue decay on the performance of SVNN, two sets of experiments were performed. In the first set of experiments a SVNN was trained by minimizing (27) without the first term; therefore, this model was named SVNN À λ min . In the second set of experiments a SVNN was rained by minimizing (27) without the first two terms; hence, this model was named SVNN À λ min À λ max . Both models were evaluated on all the datasets of Table 1 . Moreover, it was investigated the influence of the term about class proportion on the accuracy of TNN. To do so, a TNN was trained by minimizing only the first four terms of (31); hence, this model was named TNN-C 3 . The results are summarized in Table 9 .. As regards the inductive training methods, SVNN had the best performance and in the majority of the evaluated datasets by using only 10 labeled data. In the case of the Haberman dataset, all the algorithms fail, predicting all the testing data in the same class (see the value of BER¼ 50% in Table 3 ). By using all the training data, SVNN only was less accurate than SVM in the Text dataset (see Table 5 ). We believe that this fact is due to the highdimensional feature space of Text dataset, since such fact can favor linear classifiers, such as the linear SVM. As regards the transductive training methods, TSVM and TSVMrbf predicted all the testing data of the UCI datasets in the majority class when using only 10 labeled data, i.e. the constraint on the predicted class proportion was violated (see the value of BER ¼50% in the first three rows of Table 4 ). Therefore, TNN was the best approach for all the datasets, excepting the Text dataset, for which the linear TSVM was the best approach. By using all the training data, TNN had the best values of accuracy, BER, and PPV in five of the seven datasets.. As regards the training and testing time, SVM was, in most of the experiments, less expensive in training than the proposed methods; however, the testing time reveals the main advantage of SVNN and TNN, which can perform nonlinear classification a few hundred times faster than SVM with nonlinear kernels, as can be seen, for instance, in the fifth row of Table 7 . In this case, TSVM has 231 support-vectors, while the TNN has only two hidden neurons; therefore, taking into account that Digit1 dataset has 241 attributes, from the models (1) and (19) it is possible to realize that the decision function of TSVM requires the calculation of 56 133 products, 55 672 sums, and 232 nonlinear functions, while the decision function of TNN only requires the calculation of 484 products, 487 sums, and 2 nonlinear functions. Such fact is especially relevant in applications such as on-the-fly object detection, in which each image frame has to be scanned by a sliding window, generating several thousands of cropped images to be classified.. By comparing Tables 9 and 5 , it is possible to verify the positive influence of eigenvalue decay on the performance of SVNN. Table 9 also reveals the importance of the term about class proportion on the performance of TNN. Note that TNN-C 3 is unsatisfactory in classifying the first two datasets, i.e. TNN predicted all the testing data of datasets Breast Cancer and Haberman in the majority class (see the value of BER¼ 50% in the last cells of the first two rows of Table 9 ).

O O O O O O B I B O O O O O O O O O O O O O O B O B B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Knowledge mining sensory evaluation data: genetic programming, statistical techniques, and swarm optimization Abstract Knowledge mining sensory evaluation data is a challenging process due to extreme sparsity of the data, and a large variation in responses from different members (called assessors) of the panel. The main goals of knowledge mining in sensory sciences are understanding the dependency of the perceived liking score on the concentration levels of flavors' ingredients, identifying ingredients that drive liking, segmenting the panel into groups with similar liking preferences and optimizing flavors to maximize liking per group. Our approach employs (1) Genetic programming (symbolic regression) and ensemble methods to generate multiple diverse explanations of assessor liking preferences with confidence information; (2) statistical techniques to extrapolate using the produced ensembles to unobserved regions of the flavor space, and segment the assessors into groups which either have the same propensity to like flavors, or are driven by the same ingredients; and (3) two-objective swarm optimization to identify flavors which are well and consistently liked by a selected segment of assessors. In this contribution, we knowledge mine sensory evaluation data collected for the purposes of flavor research and design. Extracting flavor information from sensory evaluation data has scientific and commercial value. Our particular experimental dataset is collected for hedonic flavor evaluation. See Fig. 1 . Each sample (or query) supplied to the assessor is controlled by varying its flavoring ingredients. A hedonic response is obtained by asking an assessor to sense (e.g. smell, taste, view) something (e.g. a food product) and report how much he or she likes its flavor. The assessor reports how much he or she likes the flavor using 9 distinct language categories which range from ''like extremely'' to ''dislike extremely'' with ''neutral'' in the middle. These categoric responses are mapped to numeric scores per Fig. 2a .. In this contribution, we deal with a hedonic experiment dataset provided by Givaudan Flavors Inc. Research scientists at Givaudan prepared the experimental protocol, pre-screened the assessors, and conducted the experiment prior to our involvement in the project. Using a designed experiments approach, they prepared 36 different flavors. Each flavor is a composition of the same seven ingredients k 1 ; k 2 ; . . .; k 7 ; called keys, at different concentrations. The ranges of concentrations of different ingredients is significant: some range between [0, 200] , others between [0, 20] . Levels chosen for each ingredient in the experiment are either extrema or centers of each range. Each assessor, s, evaluates flavor k j 2 R 7 using 9-value language hedonic category scale and by assigning a liking score (LS) to it (as in the top of Fig. 1 ). Each of the 36 different flavors was evaluated by 69 assessors.. The intent of such a hedonic experiment is to facilitate understanding of which ingredients drive liking of a target population (assumed to be represented by the assessor panel), the consistency of liking preferences and to deliver insight into how to design or identify flavors that most consumers would consistently like. To extract information that provides this understanding is demanding because any dataset, ours being just one example, poses numerous technical challenges: The resulting data is best visualized to recognize the phenomenon we have to address. Figure 2b shows the responses for the 40 flavors for the 69 assessors. For 19 flavors, the responses vary across the complete range from 1 to 9. For most of the remaining flavors, the range is 7 or 8. This implies that assessors do not agree in their hedonic judgement. More technically, it implies that there can be no one single model that accurately describes the entire dataset. Instead, it is more desirable to construct a model for each assessor. 2. The data is extremely sparse The risk of rapid sensory exhaustion drives hedonic experiments to present an extremely small number of samples to an assessor. Our dataset is, in fact, only somewhat exemplary with respect to its sparsity 1 . It contains more flavor samples than usual but this amounts to only 36 flavors per assessor! Given the 7 dimensional space and range of each key, this is a very, very small fraction of samples of the flavor space. How can such a small fraction be used to predict out of sample behavior if we model an assessor? How might analytic methods be validated for accuracy? Whatever the challenge such a small sample size presents, it must be managed because modeling the entire panel is impossible due to the variation on the repeated measures. 3. Ingredients interact hedonically in a non-linear manner An additional complexity of sensory evaluation knowledge mining is the non-linear nature of the relationship between ingredients and a hedonic response. If the volumes of all ingredients in a flavor are doubled, the hedonic response will not necessarily stay the same, it might drastically change. This raises a question: Could complex models be potentially necessary to accurately model an assessor's hedonic function? 4. Assessors may use the hedonic scale differently An inquiring observer might wonder ''What does the scale mean for each assessor?'' Intuitively, people may not use the same scale to describe liking. When considering combining evaluations from many assessors, ''how is it possible to ensure that the final outcome is scale invariant?'' 5. Designing well-liked flavors Moskowitz has stated that ''taste and smell, the chemical senses, are prime examples of inter-panelist differences, especially in terms of the hedonic tone (liking/disliking)'' [19] . Continuing, Moscowitz remarks that ''panelist to panelist differences are simply too great to ignore as just an inconvenience of the scientific quest.'' In view of a business goal of designing flavors that many people like a lot, this phenomenon in the dataset logically raises questions such as: ''Can one flavor satisfy everybody? What is a realistic liking score for a flavor that satisfies everybody?'' In this paper, we presented an approach to model a liking score function for assessors employed in sensory evaluation. A GP based symbolic regression methodology was used to generate all plausible models that explain the given data. From these plausible models, a subset was selected that are diverse and that explain data analysis using model ensembles. In this new space, we respect the evidence that the response and explanatory variable relationship differs among assessors and exploit rather than inaccurately average the differences. The information of variable importance facilitates model similarity clustering on this basis and efficient sensitivity analysis. Ensemble archives of Pareto genetic programming experiments furnish the model sets for variable important analysis that can be driven by presence or fitness weighting. The standard GP approach for variable selection, which analyses variable presence in just one successful solution, does not work in this context because the variable importance statistics of one model are not reliable. We identified 6 distinct groups in our panel using multiple models per assessors. The flavor liking of members of one group are driven by the same ingredients.. Our methodology postpones decision making regarding a model, a prediction and a decision boundary until the very end. For an unseen flavor, all the models are consulted and with minor trimming, their predictions are fit to a probability density function. Finally, as the macro-level behavior emerges and more is known about the assessors, decision boundaries can be rationally imposed on this probability space to allow their segmentation. Our approach allowed us to robustly identify segments in the panel based on their propensity to like the flavor space. We conjecture from our results that there are similar potential benefits across any sparse, repeated measure dataset.. We then integrated the ensemble of liking score functions per assessor into two multi-objective algorithms that identify the flavors that maximize (a) an assessor's liking score, or (b) a group of assessors' liking scores. We used a particle swarm optimization algorithm for this. We observed that for an individual assessor, the method identified flavors that were extremely liked according to the hedonic scale. However, when the flavors are optimized for a group of assessors, the maximum liking score that we achieved was around '7' which corresponds to ''like moderately'' on the hedonic scale.. There are many choices in this knowledge mining process: e.g. what data to aggregate and thresholds such as h. They should, in general, be made by an expert on the data being modeled. A choice could depend on exogenous goals like market targeting. For example, one could decide to use average ratings of the panel. This would allow them to design flavors that maximize the liking scores according to this information. In this example, strong inter-assessor differences contra-indicate this approach. We observe that all variables are important for modeling the liking score of the entire panel and that there exist fundamental differences in the driving variables among individual panelists. This implies that an approach of designing flavors for the entire panel is likely to generate designs that will be suited to a broad population with a lesser degree of liking. Alternatively, if it is affordable to segment the panel into multiple segments and design flavors that satisfy these smaller segments, each resulting design would have higher likability inside a segment but less suitability across the broad population. The advantage of our approach is that all analysis decisions are postponed until the moment when the decision trade-offs become clear to the domain expert. To understand the trade-offs, the domain experts have access to efficient sensitivity analysis methods which will allow them to finally identify the directions in which the liking scores of panelists are steered by important keys.. We are enthusiastic about the results, primarily because they confirm that genetic programming symbolic regression methodology has evolved into a mature field capable of routinely solving real-world problems. In this case study, genetic programming allowed us to decompose a seemingly unsolvable problem (few samples with multiple responses of high variation) into a sequence of solvable problems generating insights at each step. The most exciting feature of the study is its efficiency. The complete analysis when automated takes a night (or a lot less if multiple cores are available). This, in combination with flavor optimization opens up opportunities for new on-line protocols of flavor design, generating new insights in days instead of months. Additionally, panel segmentation, derived on the basis of liking being influenced by the same ingredients in the same direction, will allow a clearer understanding of the hedonic responses to a product suite. When affordable, it may enable the development of products for particular segments leading to higher consumer satisfaction.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Effective Domain Partitioning for Multi-Clock Domain IP Core Wrapper Design under Power Constraints SUMMARY The rapid advancement of VLSI technology has made it possible for chip designers and manufacturers to embed the components of a whole system onto a single chip, called System-on-Chip or SoC. SoCs make use of pre-designed modules, called IP-cores use of pre-designed modules, called IP-cores, which provide faster design time and quicker time-to-market. Furthermore, SoCs that operate at multiple clock domains and very low power requirements are being utilized in the latest communications, networking and signal processing devices. As a result, the testing of SoCs and multi-clock domain embedded cores under power constraints has been rapidly gaining importance. In this research, a novel method for designing power-aware test wrappers for embedded cores with multiple clock domains is presented. By effectively partitioning the various clock domains, we are able to increase the solution space of possible test schedules for the core. Since previous methods were limited to concurrently testing all the clock domains, we effectively remove this limitation by making use of bandwidth conversion, multiple shift frequencies and properly gating the clock signals to control the shift activity of various core logic elements. The combination of the above techniques gains us greater flexibility when determining an optimal test schedule under very tight power constraints. Furthermore, since it is computationally intensive to search the entire expanded solution space for the possible test schedules, we propose a heuristic 3-D bin packing algorithm to determine the optimal wrapper architecture and test schedule while minimizing the test time under power and bandwidth constraints. The recent popularity of advanced technologies such as broadband Internet, 3-G cellular phones and high-speed workstations is due to many factors, one of which is the rapid advancement in the design and production of VLSI chips. More importantly, it has now become possible to put entire systems onto a single chip which is commonly known as System-on-Chip (SoC). Currently, SoCs are widely used in devices intended for telecommunications, networking and digital signal processing. Moreover, they are increasingly being utilized in mobile on-the-field devices, which increase the demand for highly-reliable, defect-free chips that have very low power requirements.. To ensure that SoCs and other VLSI chips operate as intended, testing must be conducted per chip. As production capabilities improve, clock-rates rise exponentially and transistor density increases dramatically, the cost of testing newer VLSI chips also becomes higher. More specifically, the increased complexity of the circuitry in SoCs also means an increase in the amount of test data, which usually results in longer test application time. Furthermore, test access becomes a problem since the cores cannot be directly accessed from the I/O pins of the chip. Moreover, modern IP cores operate at various frequencies internally, which have advantages such as reduced power and silicon area. These multiclock domain cores present clock skew and at-speed testing problems. Clock skew problems arise from unsynchronized clock sources such as two signals of the same frequency but different clock trees that arrive out-of-sync to their destinations thereby causing data corruption. Furthermore, power consumption and heat during test has become a big issue because of high switching activity during scan-shift operations as well as the possibility of more active components than expected during normal operation.. The most common Design-for-Testability (DFT) used for SoCs with multiple cores is the design of a test data delivery method, more commonly known as Test Access Mechanism (TAM), and the use of wrappers, which enables independent core testing. The wrapper isolates a core during test and provides an interface to apply and collect test data from it. More recently, the IEEE 1500 standard for embedded core test has been approved to provide guidelines for core wrapper design and interfacing to TAMs. Furthermore, several approaches to optimize wrapper designs for single frequency embedded core testing [1] , [2] as well as wrapper and TAM co-optimization algorithms [3] , [4] , [11] , [12] have already been suggested. Still, these approaches don't directly address the problem of testing multi-clock domain cores.. To address this problem, we propose an IEEE 1500 compliant power-aware multi-clock domain core wrapper that partitions the IP core into smaller sub-groups and utilizes gated-clocks to control the start times of scan-shift operations and enable a more flexible and efficient use of the external bandwidth under a power constraint compared to previous methods. A heuristic 3-D rectangular bin packing algorithm is also introduced, which forms the basis of the proposed wrapper design method. Most at-speed multi-clock domain core testing techniques Grouping the wrapper scan chains according to their clock domains prevents the effects of clock skew during this phase. To avoid clock skew at the capture phase, the test enters this phase before or after the last bit of test data is scanned in. In the capture phase, all the scan chains are driven by their functional clocks to simulate normal operation. For multi-clock domain cores, a capture window similar to that proposed in [8] , [9] , as shown in Fig. 3 , is used during this phase and the P-vc's are activated in such a way that ensures inter-domain and intra-domain signals are prop- Fig • ( 1) where:. 2. The total power dissipation of all active P-vc's at any time t cannot exceed the maximum allowed power dis-. Since all the P-vc's become active and inactive independently, the total scan-shift time for one test pattern can be computed from the P-vc with the latest end time as shown below: the packed P-vc's while minimizing the total height. Since it has been shown that the restricted 3-D bin packing problem is NP-Hard in [12], we propose the following heuristic algorithm to solve the problem.

O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


An

efficient

single-pass

trace

compression

technique

utilizing

instruction

streams

Trace-driven

simulations

have

been

widely

used

in

computer

architecture

for

quantitative

evaluations

of

new

ideas

and

design

prototypes.

Efficient

trace

compression

and

fast

decompression

are

crucial

for

contemporary

workloads,

as

representative

benchmarks

grow

in

size

and

number.

This

article

presents

Stream-Based

Compression

(SBC),

a

novel

technique

for

single-pass

compression

of

address

traces.

The

SBC

technique

compresses

both

instruction

and

data

addresses

by

associating

them

with

a

particular

instruction

stream,

that

is,

a

block

of

consecutively

executing

instructions.

The

compressed

instruction

trace

is

a

trace

of

instruction

stream

identifiers.

The

compressed

data

address

trace

encompasses

the

data

address

stride

and

the

number

of

repetitions

for

each

memoryreferencing

instruction

in

a

stream,

ordered

by

the

corresponding

stream

appearances

in

the

trace.

SBC

reduces

the

size

of

SPEC

Dinero

instruction

and

data

address

traces

from

to

times,

outperforming

the

best

trace

compression

techniques

presented

in

the

open

literature.

SBC

can

be

successfully

combined

with

general-purpose

compression

techniques.

The

combined

SBC-gzip

compression

ratio

is

from

to

and

the

compression

ratio

is

from

to

Moreover,

SBC

outperforms

other

trace

compression

techniques

when

both

decompression

time

and

compression

time

are

considered.

This

article

also

shows

how

the

SBC

algorithm

can

be

modified

for

hardware

implementation

with

very

modest

resources

and

only

a

minor

loss

in

compression

ratio.

Software

simulations

have

been

a

vital

tool

in

computer

architecture

for

quantitative

evaluations

of

new

ideas

and

design

prototypes.

Both

industry

and

academia

rely

extensively

on

simulation

because

it

is

the

easiest

and

least

expensive

way

to

characterize

and

explore

design

space

[Skadron

et

al.

A

common

simulation

technique

is

trace-driven

simulation,

where

the

input

to

the

simulator

is

a

trace

of

relevant

events,

collected

during

execution

of

a

realistic

workload.

Though

execution-driven

simulators

may

provide

faster

simulation

and

more

flexibility,

they

are

often

not

readily

available

for

a

computer

system

of

interest.

Building

a

full-scale

execution-driven

simulator

is

often

impractical

and

unnecessary,

and

certainly

it

is

very

expensive.

Moreover,

in

performance-tuning

efforts

for

real-world

server

applications,

recreating

conditions

and

inputs

on

an

execution-driven

simulator

is

often

impossible.

Consequently,

program

execution

traces

for

trace-driven

simulation

remain

an

important

resource

for

computer

engineers..

In

the

last

decade,

many

research

efforts

have

been

dedicated

to

trace

issues,

such

as

trace

collection,

reduction,

and

processing

[Uhlig

and

Mudge

Depending

on

its

purpose,

a

trace

can

contain

different

types

of

information.

For

example,

control

flow

analysis

needs

only

a

trace

of

the

executed

basic

blocks

or

paths.

Cache

studies

require

address

traces,

and

more

detailed

processor

simulations

also

need

instruction

words.

Branch

predictors

can

be

evaluated

using

traces

with

only

branch-relevant

information,

such

as

branch

and

target

addresses

and

branch

outcome;

ALU

unit

simulations

require

operand

values.

For

example,

a

Dinero

trace

record

consists

of

the

address

of

the

memory

reference

and

the

reference

type

(read,

write,

or

instruction

fetch)

[Edler

and

Hill

BYU

traces

include

additional

information,

such

as

the

size

of

the

data

transfer

and

processor

ID

[Thornock

and

Flanagan

ATUM

traces

(Address

Tracing

Using

Microcode)

also

include

the

process

ID,

and

encompass

information

about

system

activity,

such

as

mapping

between

physical

and

virtual

memory

at

each

translation

look-aside

buffer

miss

[Agarwal

et

al.

An

IBS

trace

record

contains

the

operation

code

and

the

user/kernel

indicator

[Uhlig

et

al.

Traces

collected

using

the

pixie

tool

differentiate

between

records

for

load/store

memory

references

of

different

size,

such

as

byte,

double,

or

word

[Smith

To

efficiently

store,

transfer,

and

use

even

a

small

collection

of

traces,

the

traces

must

be

compressed

as

much

as

possible.

Each

new

generation

of

the

industry-recognized

SPEC

benchmark

suite

has

a

longer

run-time,

larger

resource

requirements,

and

a

larger

set

of

benchmarks

[SPEC

The

number

of

executed

instructions

in

SPEC

benchmarks

with

reference

input

sets

ranges

from

to

billion

[Cantin

and

Hill

If

each

benchmark

is

executed

with

only

one

reference

input

set,

the

sum

of

the

executed

instructions

is

about

trillion.

We

would

need

almost

terabytes

of

disk

space

to

store

these

traces,

assuming

trace

records..

An

effective

trace

compression

technique

is

lossless

(i.e.,

it

does

not

introduce

any

errors

into

the

simulation),

has

a

high

compression

factor,

and

has

a

small

decompression

overhead.

Although

traditional

compression

techniques,

such

as

the

Ziv-Lempel

algorithm

[Ziv

and

Lempel

used

in

the

gzip

utility

or

Burroughs-Wheeler

transformation

[Burrows

and

Wheeler

used

in

the

utility,

offer

a

good

compression

ratio,

even

better

compression

is

possible

when

the

specific

nature

of

redundancy

in

traces

is

taken

into

account.

Better

compression

techniques

not

only

enable

more

representative

input

and

faster

execution

of

trace-driven

simulation,

but

also

help

in

other

uses

of

traces,

such

as

the

dynamic

analysis

of

program

behavior,

debugging,

and

daily

system

maintenance

[Zhou

and

Smith

This

article

introduces

Stream-Based

Compression

(SBC),

a

new

method

for

single-pass

compression

of

address

traces

and

various

extended

trace

formats.

The

SBC

algorithm

relies

on

extracting

instruction

streams.

An

instruction

stream

is

defined

as

a

sequential

run

of

instructions,

from

the

target

of

a

taken

branch

to

the

first

taken

branch

in

the

sequence.

A

stream

table

created

during

compression

encompasses

all

relevant

information

about

streams,

such

as

the

starting

address,

stream

length,

and

instruction

types.

All

instructions

from

a

stream

are

replaced

by

its

index

in

the

stream

table,

creating

a

trace

of

instruction

streams..

SBC

features

an

efficient

on-line

algorithm

for

compression

of

data

address

references.

Unlike

instruction

addresses,

data

addresses

for

a

memoryreferencing

instruction

rarely

stay

constant

during

program

execution,

but

they

can

have

a

regular

stride.

The

SBC-compressed

data

address

trace

encompasses

a

data

address

stride

and

the

number

of

repetitions

for

each

memoryreferencing

instruction

in

a

stream.

A

change

of

the

data

address

stride

results

in

another

record

in

the

compressed

trace.

The

records

are

ordered

by

the

corresponding

stream

appearances

in

the

original

trace..

In

this

article,

we

also

show

how

a

slightly

modified

SBC

algorithm

can

be

used

in

a

hardware

resource

called

a

trace

compressor

that

will

allow

on-line

trace

compression

with

only

modest

resource

usage.

Such

a

new

resource

can

be

very

useful

for

system

debugging

and

verification,

one

of

the

most

significant

problems

in

emerging

SOCs

(Systems-On-the-Chip)

where

physical

processor

pins

are

not

available

[Fisher

et

al.

The

proposed

algorithm

achieves

a

very

good

compression

ratio

and

decompression/compression

time

for

instruction

and

data

address

traces,

yet

it

is

straightforward

to

implement

and

does

not

require

code

augmentation

or

lengthy

multi-pass

trace

analysis.

Furthermore,

SBC

can

be

successfully

combined

with

general

compression

algorithms,

such

as

Ziv-Lempel

or

BurroughsWheeler..

We

evaluate

the

proposed

technique's

efficiency

by

measuring

compression

ratio,

decompression,

and

compression

time

and

comparing

its

performance

versus

the

best

existing

techniques

for

trace

compression,

such

as

PDATS

[Johnson

et

al.

TCGEN

and

LBTC

[Luo

and

John

We

also

consider

these

techniques

when

they

are

combined

with

gzip

and

A.

Milenković

and

M.

Milenković

We

use

full

instruction

and

data

address

traces

[Edler

and

Hill

of

SPEC

benchmark

programs

as

the

input.

Detailed

experimental

analysis

shows

that

SBC

outperforms

other

techniques,

achieving

a

good

balance

between

compression

ratio

and

decompression

time.

Using

the

trace

compression

ratio

of

the

sum

of

all

traces

as

a

metric,

SBC

reduces

the

trace

size

times,

versus

times

reduction

with

PDATS,

with

TCGEN,

and

with

LBTC.

When

compression

techniques

are

combined

with

gzip,

the

compression

ratio

with

SBC.gz

is

versus

with

PDATS.gz,

with

TCGEN.gz,

and

with

LBTC.gz.

Additional

compression

with

reduces

the

trace

size

even

more:

the

compression

ratio

with

is

with

with

and

with

Although

has

a

higher

compression

ratio

than

for

all

but

five

benchmarks,

has

from

to

times

shorter

decompression

time.

In

addition,

we

show

that

SBC-compressed

traces

provide

the

shortest

simulation

time

with

a

real

cache

memory

hierarchy

simulator..

The

rest

of

this

article

is

organized

as

follows:

The

next

section

gives

an

overview

and

a

new

classification

of

the

existing

techniques

for

trace

compression.

Section

explains

stream-based

compression

and

decompression

processes

and

discusses

implementation

of

the

SBC

algorithm

in

hardware.

Section

evaluates

the

effectiveness

of

SBC

versus

other

trace

compression

techniques,

comparing

compression

ratio,

compression

time,

and

decompression

time.

The

last

section

gives

concluding

remarks.

The

main

contributions

of

this

article

are

the

following:.

-A

single-pass

trace

compression

technique,

SBC.

In

the

SBC

algorithm,

an

instruction

stream

is

replaced

by

its

index

from

the

stream

table.

Data

addresses

are

linked

to

a

corresponding

instruction

stream

and

compressed

using

an

efficient

online

algorithm

that

recognizes

regular

strides.

This

technique

achieves

an

optimal

balance

between

the

most

important

compression

requirements:

high

compression

ratio

and

fast

decompression

time.

The

SBC

website,

includes

SBCcompressed

SPEC

traces

and

utility

programs.

-A

modified

SBC

with

fixed-size

resources,

SBChw,

suitable

for

hardware

implementation.

The

SBChw

technique

achieves

a

compression

ratio

comparable

to

that

of

SBC

with

very

modest

resources,

by

using

a

finite-size

stream

cache

instead

of

an

unbounded

stream

table,

and

limiting

the

maximal

stream

length

and

the

number

of

loads/stores

in

a

stream.

-Evaluation

of

SBC

versus

the

best

existing

single-pass

lossless

compression

techniques,

PDATS,

LBTC,

and

TCGEN.

SBC

outperforms

all

other

techniques

when

compression

ratio,

decompression

time,

and

compression

time

are

all

taken

into

consideration.

Without

additional

compression

by

gzip

or

SBC

compresses

better

than

all

other

techniques,

outperforming

the

next

best

technique,

LBTC,

times.

SBC

also

has

the

highest

compression

ratio

when

combined

with

gzip,

for

all

but

two

considered

applications.

When

combined

with

it

achieves

a

compression

ratio

comparable

to

that

of

Though

has

the

best

overall

compression

ratio,

it

has

a

significant

decompression

overhead.

Evaluation

of

decompression

time

in

two

settings,

with

an

optimized

decompression

program

and

a

real

simulator,

shows

that

SBC

traces

decompress

much

faster

than

TCGEN

traces.

With

the

custom

decompression

program,

SBC.gz

and

are

and

times

faster

than

respectively.

Simulation

with

the

DineroIV

cache

memory

simulator

finishes

times

faster

with

SBC.gz

than

with

and

times

faster

with

-Analysis

of

compressibility

of

instruction

and

data

trace

components

for

all

evaluated

techniques,

as

well

as

sensitivity

of

the

SBC

compression

ratio

to

the

data

address

FIFO

buffer

size..

A.

Milenković

and

M.

Milenković

-A

novel

classification

of

lossless

trace

compression

techniques,

based

on

the

method

used

for

compression

of

data

and

instruction

components..

As

our

analysis

indicates,

future

compression

techniques

should

concentrate

on

the

data

trace

component,

since

it

makes

up

over

of

the

SBC

compressed

traces.

Another

avenue

to

be

explored

is

the

extension

of

the

SBC

trace

format

for

other

types

of

traces.

An

additional

benefit

of

the

SBC

technique

is

the

separation

of

the

stream

trace

file

from

the

rest

of

the

compressed

data.

Hence,

various

statistics

such

as

the

number

of

executed

instructions

of

a

particular

type,

number

of

streams,

and

dynamic

distribution

of

streams

can

be

obtained

by

processing

only

the

stream

table

file

instead

of

the

whole

trace.

The

stream

table

file

can

also

include

information

about

basic

blocks

in

a

stream

and

a

stream

frequency

field.

In

this

section,

we

first

describe

our

evaluation

environment

and

determine

the

feasibility

of

the

SBC

technique

using

stream

statistics.

For

all

evaluated

techniques,

we

compare

the

trace

compression

ratio,

compression

and

decompression

times,

and

simulation

speedup.

We

also

evaluate

the

compression

ratio

of

SBC

modified

for

hardware

implementation.

Finally,

we

discuss

the

compressibility

of

instruction

and

data

components.

We

can

broadly

classify

traces

in

two

categories.

The

first

category

encompasses

traces

that

have

information

only

about

the

instruction

flow:

traces

of

instruction

addresses,

basic

blocks,

executed

branches,

or

procedure

calls.

The

traces

in

the

second

category

also

include

records

with

highly

variable

values

such

as

data

addresses

or

operand

values.

Consequently,

trace

compression

techniques

can

also

be

divided

into

two

groups:

techniques

targeting

only

instruction

flow

traces,

and

those

targeting

traces

including

both

instruction

and

data

information.

Figure

shows

this

classification

for

lossless

trace

compression

techniques..

The

underlying

idea

for

several

instruction

trace

compression

techniques

is

to

replace

an

execution

sequence

with

its

identifier.

For

example,

in

the

whole

program

path

technique

(WPP),

a

program

is

instrumented

to

produce

a

trace

of

acyclic

paths

[Larus

Acyclic

paths

are

then

compressed

using

a

modified

Sequitur

algorithm

[Nevill-Manning

and

Witten

WPP

yields

a

very

good

compression

ratio,

and

it

is

convenient

for

certain

types

of

analysis,

such

as

finding

the

most

frequently

executed

paths.

The

downside

is

that

it

requires

the

code

to

be

instrumented,

so

it

is

not

directly

applicable

for

already-recorded

traces.

Time-stamped

WPP

[Zhang

and

Gupta

enables

fast

access

to

the

trace

of

a

particular

function:

WPP

is

broken

into

path

traces

corresponding

to

individual

function

calls,

and

all

path

traces

for

one

function

are

stored

together

as

a

block.

Another

compression

technique

replaces

an

N

-tuple

of

original

trace

records

with

its

identifier

from

the

N

-Tuple

record

table

[Milenković

et

al.

The

reduced

trace

is

then

compressed

using

gzip.

The

redundancy

in

a

trace

is

better

exposed

to

gzip,

so

this

technique

achieves

up

to

times

better

compression

than

with

gzip

alone,

for

traces

of

branch

instruction

records

and

N

as

small

as

In

a

trace

that

includes

instruction

words,

one

can

replace

the

most

frequent

trace

records

with

their

identifiers

in

the

dictionary

(PDI

[Johnson

et

al.

Another

option

is

to

replace

a

sequence

of

repeating

trace

records

by

the

corresponding

repetition

count.

Hamou-Lhadj

and

Lethbridge

propose

one

such

technique

for

traces

of

procedure

calls:

a

trace

is

first

preprocessed

to

replace

repeating

sequences

of

calls

with

a

number

of

repetitions,

and

then

it

is

represented

as

an

ordered

labeled

tree,

where

nodes

are

procedure

calls

and

tree

levels

correspond

to

nesting

levels.

In

a

third

phase,

the

tree

is

transformed

into

a

dynamic

acyclic

graph..

QPT

is

a

tracing

and

compression

technique

that

records

only

information

about

significant

events

[Larus

A

QPT

trace

includes

only

transitions

between

basic

blocks

where

a

program

chooses

between

alternative

paths,

and

only

those

transitions

that

are

not

part

of

a

maximum

spanning

tree

of

a

control

flow

graph.

The

actual

instruction

addresses

can

be

regenerated

using

the

control

flow

graph

and

a

trace

of

transitions..

Traces

including

data

information

are

often

full

address

traces,

that

is,

instruction

and

data

address

traces.

The

simplest

way

to

reduce

the

size

of

an

address

trace

is

to

replace

an

address

with

the

offset

from

the

last

address

of

the

same

type

(instruction

reference,

data

read,

or

data

write

reference)

[Samples

This

single-pass

algorithm

is

called

Mache.

The

packed

differential

address

and

time

stamp

(PDATS)

algorithm

takes

the

Mache

approach

one

step

further

[Johnson

et

al.

PDATS

also

stores

address

offsets

between

successive

references

of

the

same

type,

but

the

records

in

the

trace

of

offsets

can

have

variable

lengths,

specified

in

a

one-byte

record

header,

and

an

optional

repetition

count.

For

full

address

traces

including

instruction

words,

PDATS

can

be

combined

with

a

dictionary

approach

into

PDI.

PDATS

and

Mache

have

very

small

compression/decompression

overhead,

but

they

do

not

take

into

account

the

underlying

structure

of

the

executed

program.

Luo

and

John

propose

locality-based

online

trace

compression

(LBTC)

that

targets

more

complex

trace

records.

Information

about

an

executed

instruction

is

kept

in

a

small

direct-mapped

compression

cache.

A

compressed

trace

record

consists

of

a

record

type

(instruction

or

data),

a

cache

hit/miss

flag,

and

possibly

an

offset

from

the

previous

record

of

the

same

type.

Other

fields

of

the

original

trace

record,

such

as

instruction

word

or

virtual

address,

are

emitted

to

the

compressed

trace

only

in

the

case

of

a

cache

miss.

This

technique

keeps

only

the

last

data

address

together

with

information

about

the

corresponding

memory

referencing

instruction.

If

a

data

address

is

repeated,

a

cache-hit

flag

is

emitted;

otherwise,

the

compressed

trace

includes

the

offset

from

the

previous

data

address

in

the

trace,

similar

to

the

PDATS

technique..

Another

approach

is

to

link

information

about

the

data

addresses

with

a

corresponding

loop

[Elnozahy

In

the

first

pass,

loops

in

the

trace

are

detected

using

a

control

flow

analysis

technique.

In

the

second

pass,

data

address

references

inside

each

loop

are

classified

as

chaotic,

constant,

or

loop

varying,

that

is,

with

a

constant

offset

between

loop

iterations.

Constant

and

loop-varying

addresses

need

to

be

encoded

only

once

in

the

compressed

trace,

but

all

chaotic

addresses

must

be

stored

separately.

Control

flow

analysis

to

extract

loop

information

can

be

avoided

if

a

program

is

instrumented

before

tracing

[DeRose

et

al.

However,

the

limitation

of

this

technique

is

that

the

iteration

count

for

inner

loops

must

be

constant..

Information

about

data

addresses

can

also

be

linked

to

an

instruction

block

[Pleszkun

For

each

memory-referencing

instruction

in

an

instruction

block,

the

possible

data

offsets

and

numbers

of

repetitions

are

recorded.

This

technique

may

have

very

large

memory

requirements,

since

information

about

all

possible

data

address

offsets

for

one

load

or

store

instruction

is

kept

in

a

linked

list.

However,

for

data

references

without

a

fixed

offset

or

with

a

large

number

of

different

offsets,

this

approach

may

not

be

feasible.

Pleszkun

also

proposes

compression

for

the

instruction

component:

an

intermediate

compressed

trace

encompasses

the

identifiers

of

successors

for

each

instruction

block

and

the

number

of

repetitions.

During

a

second

pass,

static

basic

blocks

are

fused

into

larger,

dynamic

basic

blocks..

Data

address

traces

can

be

regenerated

by

using

so-called

abstract

execution

[Larus

For

each

data

address,

a

set

of

instructions

computing

that

address

is

identified,

and

these

instructions

are

re-executed

during

trace

regeneration.

Instructions

are

classified

as

easy,

calculating

a

constant

value;

hard,

computing

a

value

that

depends

on

previously

computed

values;

and

impossible,

with

dependencies

too

complex

to

recalculate.

Instruction

addresses

are

regenerated

using

the

QPT

technique.

Abstract

execution

requires

access

to

the

source

code,

and

it

generates

a

significant

overhead

during

trace

decompression.

A

similar

technique

is

used

for

tracing

on

a

shared

memory

multiprocessor

[Eggers

et

al.

Traces

can

also

be

regenerated

using

value

predictors

(VPC

[Burtscher

and

Jeeradit

[Burtscher

and

TCGEN

The

latest

implementation,

TCGEN,

automatically

generates

optimized

compression

and

decompression

code,

featuring

a

configurable

set

of

value

predictors

for

each

component

of

a

trace

record.

During

compression,

each

component

of

a

trace

record

is

fed

to

a

separate

set

of

value

predictors,

indexed

by

the

instruction

addresses.

Each

trace

component

is

compressed

into

two

subtraces.

If

one

of

the

predictors

in

a

set

is

able

to

predict

the

component

value,

the

identifier

of

that

predictor

is

written

to

the

value

predictor

code

subtrace.

If

the

value

is

mispredicted

by

all

predictors,

it

is

written

into

the

mispredicted

value

subtrace,

and

a

reserved

code

is

written

into

the

predictor

code

subtrace.

Each

subtrace

is

further

compressed

using

This

technique

has

a

very

good

compression

ratio,

but

it

requires

a

relatively

long

decompression

time.

O O O B I O O B I O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Deductive rules in holographic reduced representation Holographic reduced representation is based on a suitable distributive coding of structured information in conceptual vectors, which elements satisfy normal distribution N(0,1/n). Existing applications of this approach concern various models of associative memory that exploit a simple algebraic operation of scalar product of distributed representations to measure an overlap between two structured concepts. This paper describes an inference process based on the rules modus ponens and modus tollens. r A modern view of the relation between brain and mind is based on the neuroscience paradigm [9] , according to which the architecture of the brain is determined by connections between neurons, their inhibitory or excitatory character and also by the strength of the connections. Human brain displays a great plasticity, synapses are perpetually formed (but also deleted) during a learning process. It can be stated, that an ability of brain to perform not only cognitive activities, but also to serve as memory and control center for our motoric activities, is fully encoded by its architecture. The metaphor of the brain as a computer should be therefore reformulated: the computer in the metaphor should be specified as a parallel distributed computer (containing many billions of neurons, elementary processors interconnected into a complex neural network).. A program in such a parallel computer is directly encoded in the architecture of the neural network, i.e. human brain is a single-purpose parallel computer represented by a neural network, which cannot be reprogrammed without a change of its architecture.. It follows that the mind and the brain create one integral unit, which is characterized by a complementary dualism. In this approach, the mind is a program carried out by the brain. This program is specified by architecture of the distributed neural network representing the brain. The brain and the mind are two different aspects of the same object:. (1) When talking about the brain, we refer to a ''hardware'' structure, biologically determined by neurons and their synaptic connections (formally represented by a neural network). (2) When talking about the mind, we refer to cognitive and other similar activities of the brain, which are carried out on a symbolic level, where the transformation of symbolic information is processed by (simple) rules.. Complementary dualism between brain and mind causes certain difficulties in the interpretation of cognitive activities of the mind. A purely neural approach to the interpretation of cognitive activities of the mind focuses on the search of neural correlates of cognitive activities (connectionism). The application of the neural paradigm to the interpretation of symbolic cognitive activities has a ARTICLE IN PRESS ''side effect'' in ''dissolving'' of these activities in their microscopic description; symbols quasi ''disappear'' in the detailed description of activities of neurons, strengths of synaptic connections, etc. On the other side, the absolute acceptance of symbolic paradigm in interpretation of cognitive activities of mind (cognitivism) ignores the fact, that mind is thoroughly embedded in brain. Purely symbolic paradigm usually leads to a conceptual sterility. An effort to explain cognitive activities of human mind only in the phenomenological terms derived from the concept of symbol is mostly fruitless. It leads to symbolic constructs (methods, algorithms, etc.) without experimental support in neuroscience.. The goal of this paper is to highlight an alternative approach, which may overcome the gap between the connectionist and cognitivist approaches to the description and interpretation of cognitive activities of the human brain [25, 27, 28, 29] . Distributed representation, where mental representations (symbols) are specified by distributed patterns of neural activities, allows to integrate connectionism and cognitivism. Formal algebraic operations can be introduced over these distributed patterns. These operations allow mathematical modeling of cognitive operations, as well as simulation of information storage and retrieval processes in memory.. We shall turn our attention to a nontraditional style of performing calculation by using distributed patterns. This approach is substantially different from classical numeric and symbolic computations and it is a suitable model tool for understanding global properties of neural networks. Such a ''neurocomputing'' is based on extensive randomly created patterns (represented by multidimensional vectors with random entries). This approach, which basic principles were formulated already at the end of the sixties [3, 10, 11, 24, 26] , was followed by Willshaw [30] , Borselino [1] , and Murdock [12, 13] , and crowned by a series of works by Tony Plate [17] [18] [19] on ''holographic reduced representation'' (HRR). Kanerva [6] [7] [8] in the middle of nineties proposed a certain alternative to HRR, which is based on randomly generated binary vectors. Rachkovskij and Kussul suggested a new representation and processing of structures with binary sparse distributed codes [22, 23] . Jane Neumann [14] [15] [16] showed that a simple gradient descent approach can be used to learn the holistic transformation of HRR from examples. The acquired knowledge can be generalized to structures containing unseen elements and to structures more complex than the training examples.. Our results as well as results of Plate [19] indicate, that transformation vector need not be constructed by a gradient optimization method, but it may be constructed by one-step constructive approach.. The goal of the present paper is to apply HRR to model simple higher level cognitive processes of reasoning by application of rules modus ponens and modus tollens in propositional and predicate logic. To keep the paper readable, in its first part we shortly reformulated basic results obtained by Plate [19] . A standard definition of an arithmetic operation k modulo n is determined as a remainder after integer division by a number n. It is necessary to comment, that the used definition of the operation k modulo n is different from this standard definition for negative numbers k. While the standard definition provides a result with a negative value, if the result is negative in our definition, then it is transformed by adding n to it.. The operation of involution satisfies the equations. A numerical implementation of the convolution can be carried out by two different approaches. The first one is based on formula (4); it offers a simple implementation, but its running time is O(n 3 ). The second one is much more sophisticated and complex and it corresponds to an application of fast Fourier transform (FFT) [21] ; its running time is O(n log n), see Fig. 1 .. We prove that an involution c Ã is roughly proportional to an inverse vector c À1 , c n c % 1. Let us study the ith component of convolution c n c. The zero-component of convolution ðc n cÞ 0 corresponds to a scalar product c Á c expressed as a sum of positive ''diagonal'' terms c 2 i , whereas other components ðc n cÞ i , for iX1, are determined by sums of ''nondiagonal'' terms c i c j , which have random signs. This observation has an important consequence that ðc n cÞ 0 is much greater than absolute values of remaining components ðc n cÞ i , for iX1, then we proved that c n c % 1, which was to be proved (see Fig. 2) .. One of the basic aspects of the holographic representation is the possibility of reconstruction of the original components, which were used for construction of convolution of two vectors. This possibility is very important, since it allows to decode the original information from the complex conceptual vectors. Reconstruction of x from c x is based on the formula proved above, c n c % 1. according to which the convolution c n with the vector c x produces the vectorx, which is similar to the original vector x,x % x. This result can be reformulated in the form. . .. where the elements of g can be regarded as normally distributed random noise with a zero mean and a standard deviation much smaller than the standard deviation of the elements of x. The overlap of the resulting vectorx with the original vector x is determined by cosine of these two vectors with real values where the inequalities result directly from the Schwartz's inequality from linear algebra. The closer this value is to its maximum value, the more similar 2 the vectorsx and x are. The Fig. 3 shows a histogram of overlaps of vectors x andx, where the most common overlap betweenx ¼ c n c x and x is around 0.7 (vectors c and x are randomly generated different conceptual vectors of dimension n ¼ 1000). It follows that the vectorsx and x are similar,x % x. Formula (12) may be also verified by the approach called the ''superposition memory''. Suppose we have a set containing p þ q randomly generated conceptual vectors, X ¼ fx 1 ; x 2 ; :::; x p ; x pþ1 ; :::; x pþq g, while poq. Using the first p vectors from X allows us to define a memory vector t as their sum A suitable distributive coding of structured information (sequence of symbols, nested relational structures, etc.) is very important in artificial intelligence and cognitive science. Holographic reduced representation offers new unconventional approach to such a coding. The used distributed representation is based on two binary operations: unary operation ''involution'' and binary operation ''convolution'' over a domain of n-dimensional randomly generated conceptual vectors, which elements satisfy normal distribution N(0,1/n). Application of this distributed representation allows us to model various types of associative memory, which are represented by a conceptual vector. It also allows us to decode a memory vector, i.e. to determine the conceptual (atomic) vectors it is composed of. Such an analysis of the memory vector is carried out by a clean-up procedure that determines from the cosine of the angle between the vectors, which of the vectors is the most similar to the memory vector. Holographic reduced representation allows measuring a similarity between two structured concepts by a simple algebraic operationscalar product of their distributed representations. This product can be very useful for modeling of processes, which search through memory to find its similar (analogical) single components. Of course, the procedure, as well as any procedure based on randomly generated vectors, may occasionally fail to provide the correct answer, but this possibility decreases with the size of the used vectors.. The main contribution of this paper is to outline a possibility, how to use the holographic reduced representation to model an inference process based on the rules modus ponens and modus tollens in propositional calculus as well as in predicate logic. This possibility is very important for a proper assessment of distributed representation (including neural networks) in artificial intelligence and cognitive science. It enables to implement higher cognitive activities, which was retained in literature mainly by symbolic localist representations.

O O O B B I B B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B I O O O O O O O O O O B I O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O