Maximally informative subspaces : nonparametric estimation for dynamical systems

Modeling complex dynamical systems is a difficult problem with a wide range of applications in prediction, discrimination, and simulation. Classical stochastic models make a number of simplifying assumptions to improve tractability (e.g. linear dynamics, Gaussian uncertainty). While such assumptions lead to algorithms which are both fast and optimal under the assumptions, there are a great many real world problems for which these assumptions are false. Recently, computational power has increased to the point where another method becomes feasible – purely example-based, or “nonparametric”, models. Yet these are limited because their computational requirements grow exponentially with the number of variables we observe about the system. For dynamical systems, in which we generally observe the past, this means that processes with any substantial past-dependence become intractable. In this thesis we present a novel dynamical system model making use of a nonparametric estimate of uncertainty, with an information-theoretic criterion for reducing the model’s required dimension while preserving as much of the predictive power in the observations as possible. To explore its behavior, we apply this technique to three dynamical systems – a “toy” nonlinear system (random telegraph waves), a real-world time series from predictive literature (Santa Fe laser data), and a more cutting-edge application (on-line signature authentication). Each of these examples demonstrates techniques for improving the model’s performance and evidence of its effectiveness.  John W. Fisher III Title: Research Scientist, Laboratory for Information and Decision Systems Maximally Informative Subspaces: Nonparametric Estimation for Dynamical Systems by Alexander T. Ihler Submitted to the Department of Electrical Engineering and Computer Science on August 9, 2000, in partial fulfillment of the requirements for the degree of Master of Science in Electrical Engineering Abstract Modeling complex dynamical systems is a difficult problem with a wide range of applications in prediction, discrimination, and simulation. Classical stochastic models make a number of simplifying assumptions to improve tractability (e.g. linear dynamics, Gaussian uncertainty). While such assumptions lead to algorithms which are both fast and optimal under the assumptions, there are a great many real world problems for which these assumptions are false. Recently, computational power has increased to the point where another method becomes feasible – purely example-based, or “nonparametric”, models. Yet these are limited because their computational requirements grow exponentially with the number of variables we observe about the system. For dynamical systems, in which we generally observe the past, this means that processes with any substantial past-dependence become intractable. In this thesis we present a novel dynamical system model making use of a nonparametric estimate of uncertainty, with an information-theoretic criterion for reducing the model’s required dimension while preserving as much of the predictive power in the observations as possible. To explore its behavior, we apply this technique to three dynamical systems – a “toy” nonlinear system (random telegraph waves), a real-world time series from predictive literature (Santa Fe laser data), and a more cutting-edge application (on-line signature authentication). Each of these examples demonstrates techniques for improving the model’s performance and evidence of its effectiveness.  John W. Fisher III Title: Research Scientist, Laboratory for Information and Decision SystemsModeling complex dynamical systems is a difficult problem with a wide range of applications in prediction, discrimination, and simulation. Classical stochastic models make a number of simplifying assumptions to improve tractability (e.g. linear dynamics, Gaussian uncertainty). While such assumptions lead to algorithms which are both fast and optimal under the assumptions, there are a great many real world problems for which these assumptions are false. Recently, computational power has increased to the point where another method becomes feasible – purely example-based, or “nonparametric”, models. Yet these are limited because their computational requirements grow exponentially with the number of variables we observe about the system. For dynamical systems, in which we generally observe the past, this means that processes with any substantial past-dependence become intractable. In this thesis we present a novel dynamical system model making use of a nonparametric estimate of uncertainty, with an information-theoretic criterion for reducing the model’s required dimension while preserving as much of the predictive power in the observations as possible. To explore its behavior, we apply this technique to three dynamical systems – a “toy” nonlinear system (random telegraph waves), a real-world time series from predictive literature (Santa Fe laser data), and a more cutting-edge application (on-line signature authentication). Each of these examples demonstrates techniques for improving the model’s performance and evidence of its effectiveness.  John W. Fisher III Title: Research Scientist, Laboratory for Information and Decision Systems