Identifying Temporal Patterns for Characterization and Prediction of Financial Time Series Events

The novel Time Series Data Mining (TSDM) framework is applied to analyzing financial time series. The TSDM framework adapts and innovates data mining concepts to analyzing time series data. In particular, it creates a set of methods that reveal hidden temporal patterns that are characteristic and predictive of time series events. This contrasts with other time series analysis techniques, which typically characterize and predict all observations. The TSDM framework and concepts are reviewed, and the applicable TSDM method is discussed. Finally, the TSDM method is applied to time series generated by a basket of financial securities. The results show that statistically significant temporal patterns that are both characteristic and predictive of events in financial time series can be identified.

Related papers:

title: DATA MINING OF MULTIPLE NONSTATIONARY TIME SERIES
abstract: A data mining method for synthesizing multiple time series is presented. Based on a single time series algorithm, the method embeds multiple time series into a phase space. The reconstructed state space allows temporal pattern extraction and local model development. Using an a priori data mining objective, an optimal local model is chosen for short-term forecasting. For the same sampling period, multiple time series embedding produces better temporal patterns than single time series embedding. The method is applied to a financial time series.

title: Using Genetic Algorithms To Find Temporal Patterns Indicative Of Time Series Events
abstract: A new framework for analyzing time series data called Time Series Data Mining (TSDM) is introduced. This framework adapts and innovates data mining concepts to analyzing time series data. In particular, it creates methods that reveal hidden temporal patterns that are characteristic and predictive of time series events. The TSDM framework, concepts, and methods, which use a genetic algorithm to search for optimal temporal patterns, are explained and the results are applied to real-world time series from the engineering and financial domains.

title: Time series data mining: identifying temporal patterns for characterization and prediction of time series events
abstract: A new framework for analyzing time series data called Time Series Data Mining (TSDM) is introduced. This framework adapts and innovates data mining concepts to analyzing time series data. In particular, it creates a set of methods that reveal hidden temporal patterns that are characteristic and predictive of time series events. Traditional time series analysis methods are limited by the requirement of stationarity of the time series and normality and independence of the residuals. Because they attempt to characterize and predict all time series observations, traditional time series analysis methods are unable to identify complex (nonperiodic, nonlinear, irregular, and chaotic) characteristics. TSDM methods overcome limitations of traditional time series analysis techniques. A brief historical review of related fields, including a discussion of the theoretical underpinnings for the TSDM framework, is made. The TSDM framework, concepts, and methods are explained in detail and applied to real-world time series from the engineering and financial domains.

title: Time series and system analysis with applications
abstract: A comprehensive, applications-oriented treatment of time series analysis. Integrates time series theory with methods of systems analysis. Clearly explains the use of ARMA forecasts and includes a complete treatment of the Box/Jenkins approach to modelling. Provides worked examples.

title: Genetic Programming Prediction of Stock Prices
abstract: Based on predictions of stock-pricesusing genetic programming (or GP), a possiblyprofitable trading strategy is proposed. A metricquantifying the probability that a specific timeseries is GP-predictable is presented first. It isused to show that stock prices are predictable. GPthen evolves regression models that produce reasonableone-day-ahead forecasts only. This limited ability ledto the development of a single day-trading strategy(SDTS) in which trading decisions are based onGP-forecasts of daily highest and lowest stock prices.SDTS executed for fifty consecutive trading days ofsix stocks yielded relatively high returns oninvestment.

title: A Probabilistic Approach to Fast Pattern Matching in Time Series Databases
abstract: The problem of efficiently and accurately locating patterns of interest in massive time series data sets is an important and non-trivial problem in a wide variety of applications, including diagnosis and monitoring of complex systems, biomedical data analysis, and exploratory data analysis in scientific and business time series. In this paper a probabilistic approach is taken to this problem. Using piecewise linear segmentations as the underlying representation, local features (such as peaks, troughs, and plateaus) are defined using a prior distribution on expected deformations from a basic template. Global shape information is represented using another prior on the relative locations of the individual features. An appropriately defined probabilistic model integrates the local and global information and directly leads to an overall distance measure between sequence patterns based on prior knowledge. A search algorithm using this distance measure is shown to efficiently and accurately find matches for a variety of patterns on a number of data sets, including engineering sensor data from space Shuttle mission archives. The proposed approach provides a natural framework to support user-customizable "query by content" on time series data, taking prior domain information into account in a principled manner.

title: A Fast and Robust Method for Pattern Matching in Time Series Databases Eamonn Keogh
abstract: The problem of finding patterns of interest in time series databases (query by content) is an important one, with applications in virtually every field of science. A variety of approaches have been suggested. These approaches are robust to noise, offset translation, and amplitude scaling to varying degrees. However, they are all extremely sensitive to scaling in the time axis (longitudinal scaling). We present a method for similarity search that is robust to scaling in the time axis, in addition to noise, offset translation, and amplitude scaling. The method has been tested on medical, financial, space telemetry and artificial data. Furthermore the method is exceptionally fast, with the predicted 2 to 4 orders of magnitude speedup actually observed. The method uses a piecewise linear representation of the original data. We also introduce a new algorithm which both decides the optimal number of linear segments to use, and produces the actual linear representation.

title: An Enhanced Representation of Time Series Which Allows Fast and Accurate Classification, Clustering and Relevance Feedback
abstract: We introduce an extended representation of time series that allows fast, accurate classification and clustering in addition to the ability to explore time series data in a relevance feedback framework. The representation consists of piece-wise linear segments to represent shape and a weight vector that contains the relative importance of each individual linear segment. In the classification context, the weights are learned automatically as part of the training cycle. In the relevance feedback context, the weights are determined by an interactive and iterative process in which users rate various choices presented to them. Our representation allows a user to define a variety of similarity measures that can be tailored to specific domains. We demonstrate our approach on space telemetry, medical and synthetic data.

title: Continuous Categories For a Mobile Robot
abstract: Autonomouas gents make frequent use of knowledgei n the form of categories -- categories of objects, human gestures, web pages, and so on. This paper describes a way for agents to learn such categories for themselves through interaction with the environment. In particular, the learning algorithm transforms raw sensor readings into clusters of time series that have predictive value to the agent. We address several issues related to the use of an uninterpreted sensory apparatus and show specific examples where a Pioneer I mobile robot interacts with objects in a cluttered laboratory setting.

title: Pattern Directed Mining of Sequence Data
abstract: Sequence data arise naturally in many applications, and can be viewed as an ordering of events, where each event has an associated time of occurrence. An important characteristic of event sequences is the occurrence of episodes, i.e. a collection of events occurring in a certain pattern. Of special interest are frequent episodes, i.e. episodes occurring with a frequency above a certain threshold. In this paper, we study the problem of mining for frequent episodes in sequence data. We present a framework for efficient mining of frequent episodes which goes beyond previous work in a number of ways. First, we present a language for specifying episodes of interest. Second, we describe a novel data structure, called the sequential pattern tree (SP Tree), which captures the relationships specified in the pattern language in a very compact manner. Third, we show how this data structure can be used by a standard bottom-up mining algorithm to generate frequent episodes in an efficient manner. Finally, we show how the SP Tree can be optimized by sharing common conditions, and evaluating each such expression only once. We present the results of an evaluation of the proposed techniques.

title: Advances in Knowledge Discovery and Data Mining
abstract: An automated data mining service offers an out-sourced, cost-effective analysis option for clients desiring to leverage their data resources for decision support and operational improvement. In the context of the service model, typically the client provides the service with data and other information likely to aid in the analysis process (e.g. domain knowledge, etc.). In return, the service provides analysis results to the client. We describe the required processes, issues, and challenges in automating the data mining and analysis process when the high-level goals are: (1) to provide the client with a high quality, pertinent analysis result; and (2) to automate the data mining service, minimizing the amount of human analyst effort required and the cost of delivering the service. We argue that by focusing on client problems within market sectors, both of these goals may be realized.

title: Predictive data mining - a practical guide
abstract: 1 What is Data Mining? 2 Statistical Evaluation for Big Data 3 Preparing the Data 4 Data Reduction 5 Looking for Solutions 6 What's Best for Data Reduction and Mining? 7 Art or Science? Case Studies in Data Mining

title: Forecasting and Time Series: An Applied Approach
abstract: An introduction to forecasting Basic statistical concepts Forecasting by using regression analysis Simple linear regression Multiple regression Topics in regression analysis Forecasting by using time series regression, decomposition methods and exponential smoothing Time series regression Decomposition methods Exponential smoothing Forecasting by using basic techniques of the box Jenkins methodology Nonseasonal, box-Jenkins models and their tentative identification Estimation, diagnostic checking and forecasting for nonseasonal box-Jenkins models An introduction to box-Jenkins seasonal modelling Forecasting by using advanced technology of the box-Jenkins methodology General box-Jenkins seasonal modelling Using the box-Jenkins methodology to improve time series regression models and to implement exponential smoothing Transfer functions and intervention models.

title: IMPROVING GENETIC ALGORITHMS PERFORMANCE BY HASHING FITNESS VALUES
abstract: This paper presents a method for improving genetic algorithm (GA) performance. Typically, zero diversity in the population's fitness values signals the stopping point for a GA. As the population evolves, diversity diminishes, causing the same chromosomes to be frequently reevaluated. For real world problems, the computational effort spent on evaluating the fitness function far exceeds that of the genetic operators. By using a hash table to store the most recently evaluated chromosomes, significant performance improvements are realized. Several examples demonstrate the improvements.

title: Applications of Fractals and Chaos
abstract: We may not be able to make you love reading, but applications of fractals and chaos will lead you to love reading starting from now. Book is the window to open the new world. The world that you want is in the better stage and level. World will always guide you to even the prestige stage of the life. You know, this is some of how reading will give you the kindness. In this case, more books you read more knowledge you know, but it can mean also the bore is full.

title: Experimental approach to nonlinear dynamics and chaos
abstract: This essential handbook provides the theoretical and experimental tools necessary to begin researching the nonlinear behavior of mechanical, electrical, optical, and other systems. The book describes several nonlinear systems which are realized by desktop experiments, such as an apparatus showing chaotic string vibrations, an LRC circuit displaying strange scrolling patterns, and a bouncing ball machine illustrating the period doubling route to chaos. Fractal measures, periodic orbit extraction, and symbolic analysis are applied to unravel the chaotic motions of these systems. The simplicity of the examples makes this an excellent book for undergraduate and graduate-level physics and mathematics courses, new courses in dynamical systems, and experimental laboratories.

title: Improving Computational Performance of Genetic Algorithms: A Comparison of Techniques
abstract: A comparison of three methods for saving previously calculated fitness values across generations of a genetic algorithm is made. These methods lead to significant computational performance improvements. For real world problems, the computational effort spent on evaluating the fitness function far exceeds that of the genetic operators. As the population evolves, diversity usually diminishes. This causes the same chromosomes to be frequently reevaluated. By using appropriate data structures to store the evaluated fitness values of chromosomes, significant performance improvements are realized. Several different data structures are compared and contrasted. This paper shows that, for different sets of genetic algorithm parameters, including selection type, population size, and level of mutation, performance improvements are realized.

title: Investment Analysis and Portfolio Management
abstract: The purpose of this book is to help you learn how to manage your money to derive the maximum benefit from what you earn. Mixing investment instruments and capital markets with the theoretical detail on evaluating investments and opportunities to satisfy risk-return objectives along with how investment practice and theory is influenced by globalization leaves readers with the mindset on investments to serve them well. The material is intended to be rigorous and empirical yet not overly quantitative. We continue with unparalleled international coverage, newly rewritten and reorganized derivatives material to be more intuitive and clearer, three additional chapters on derivatives pricing for those who want more detail, rewritten material on multifactor models of risk and return, and new CFA problems for more practice on computations concerning investment decisions. To manage money and investments, one needs to learn about investment alternatives and develop a way of analyzing and thinking about investments that will be of benefit and allow a foundation as new tools and investment opportunities become available. Reilly/Brown provide the best foundation, used extensively by professionals, organizations, and schools across the country. A great source for those with both a theoretical and practical need for investment expertise.

title: Behind the Smoke and Mirrors: Gauging the Integrity of Investment Simulations
abstract: Fund sponsors and others who must evaluate simulated investment results should carefully question the simulation process. In particular, they should ask about the data base used, the portfolio construction technique, and the assumptions the manager made about portfolio tumrover and transaction costs. The manager should be able to document construction of the simulation and defend the decisions that went into