symbolic model checking for real time systems abstract we describe finite state programs over real numbered time in a guarded command language with real valued clocks or equivalently as finite automata with real valued clocks model checking answers the question which states of a real time program satisfy a branching time specification given in an extension of ctl with clock variables we develop an algorithm that computes this set of states symbolically as a fixpoint of a functional on state predicates without constructing the state space for this purpose we introduce a calculus on computation trees over real numbered time unfortunately many standard program properties such as response for all nonzeno execution sequences during which time diverges cannot be characterized by fixpoints we show that the expressiveness of the timed calculus is incomparable to the expressiveness of timed ctl fortunately this result does not impair the symbolic verification of implementable real time programs those whose safety constraints are machine closed with respect to diverging time and whose fairness constraints are restricted to finite upper bounds on clock values all timed ctl properties of such programs are shown to be computable as finitely approximable fixpoints in a simple decidable theory

control of singularly perturbed hybrid stochastic systems we study a class of optimal stochastic control problems involving two different time scales the fast mode of the system is represented by deterministic state equations whereas the slow mode of the system corresponds to a jump disturbance process under a fundamental ergodicity property for a class of infinitesimal control systems associated with the fast mode we show that there exists a limit problem which provides a good approximation to the optimal control of the perturbed system both the finite and infinite discounted horizon cases are considered we show how an approximate optimal control law can be constructed from the solution of the limit control problem in the particular case where the infinitesimal control systems possess the so called turnpike property i e characterized by the existence of global attractors the limit control problem can be given an interpretation related to a decomposition approach

percolation thresholds critical exponents and scaling functions on planar random lattices and their duals the bond percolation process is studied on periodic planar random lattices and their duals the thresholds and critical exponents of the percolation transition are determined the scaling functions of the percolating probability the existence probability of the appearance of percolating clusters and the mean cluster size are also calculated the simulation result of the percolation threshold is pc50 333360 0001 for planar random lattices and 0 667060 0001 for the duals of planar random lattices we conjecture that the exact value of pc is 1 3 for a planar random lattice and 2 3 for the dual of a planar random lattice by taking possible errors into account the results of our critical exponents agree with the values given by the universality hypothesis by properly adjusting the metric factors on random lattices and their duals we demonstrate explicitly that the idea of a universal scaling function with nonuniversal metric factors in the finite size scaling theory can be extended to random lattices and their duals for the existence probability the percolating probability and the mean cluster size s1063 651x 99 00112 9

symbolic model checking for probabilistic timed automata probabilistic timed automata are timed automata extended with discrete probability distributions and can be used to model timed randomised protocols or fault tolerant systems we present symbolic model checking algorithms for probabilistic timed automata to verify both qualitative temporal logic properties corresponding to satisfaction with probability 0 or 1 and quantitative properties corresponding to satisfaction with arbitrary probability the algorithms operate on zones which represent sets of valuations of the probabilistic timed automaton s clocks our method considers only those system behaviours which guarantee the divergence of time with probability 1 the paper presents a symbolic framework for the verification of probabilistic timed automata against the probabilistic timed temporal logic ptctl we also report on a prototype implementation of the algorithms using difference bound matrices and present the results of its application to the csma cd and firewire root contention protocol case studies

approximation and decomposition of singularly perturbed stochastic hybrid systems this paper considers a singularly perturbed stochastic hybrid system whose state equations are governed by a stochastic switching process which is modelled as a nearly completely decomposable continuous time finite state markov chain approximate models for the slow mode subsystem are derived over the intervals of rapid switching and their accuracy the mean squared errors between the approximate models and the original systems is quantified aggregate models for the subsystem are obtained over a longer period the almost surely exponential stability of the subsystem is studied the behaviour of the fast subsystem which depends on the relative size of the perturbation parameters is analysed the decomposition of the overall system and the switching process together into slow and fast subsystems is investigated the results derived in the paper are shown to hold when the switching process is stationary and irreducible and each group of strongly interacting states is irreducible and time reversible fina

exponential bounds for discrete time singularly perturbed markov chains abstract this paper develops exponential type upper bounds for scaled occupation measures of singularly perturbed markov chains in discrete time by considering two time scale in the markov chains asymptotic analysis is carried out the cases of the fast changing transition probability matrix is irreducible and that are divisible into l ergodic classes are examined first the upper bounds of a sequence of scaled occupation measures are derived then extensions to markov chains involving transient states and or nonhomogeneous transition probabilities are dealt with the results enable us to further our understanding of the underlying markov chains and related dynamic systems which is essential for solving many control and optimization problems

analytic perturbation of generalized inverses abstract we investigate the analytic perturbation of generalized inverses firstly we analyze the analytic perturbation of the drazin generalized inverse also known as reduced resolvent in operator theory our approach is based on spectral theory of linear operators as well as on a new notion of group reduced resolvent it allows us to treat regular and singular perturbations in a unified framework we provide an algorithm for computing the coefficients of the laurent series of the perturbed drazin generalized inverse in particular the regular part coefficients can be efficiently calculated by recursive formulae finally we apply the obtained results to the perturbation analysis of the moore penrose generalized inverse in the real domain

aggregation of the policy iteration method for nearly completely decomposable markov chains a steady state optimal control problem is considered for nearly completely decomposable markov chains in order to apply the policy iteration method of r a howard dynamic programming and markov processes cambridge ma mit press 1960 a high dimensional ill conditioned system of algebraic equations must be solved in the value determination step although algorithms exist for aggregation of the steady state probability distribution problem they only provide methods for computing the cost not the dual variables using a singular perturbation approach an aggregation method for the value determination equation is developed the aggregation method is developed in three steps first a class of similarity transformations that transform the system into a singularly perturbed form is developed second an aggregation method to compute the steady state probability distribution is derived third this aggregation method is applied to the value determination step of howard s method 1960

computing cumulative measures of stiff markov chains using aggregation an aggregation method for computing transient cumulative measures of large stiff markov models is presented the method is based on classifying the states of the original problem into slow fast transient and fast current states the authors aggregate fast transient states and fast recurrent states so that an approximate value to the desired cumulative measure can be obtained by solving a nonstiff set of linear differential equations defined over a reduced subset of slow states only several examples are included to illustrate how stiffness arises naturally in actual queuing and reliability models and to show that cumulative measures provide a better characterization of the time dependent system behavior

on probabilistic timed automata we propose a model of probabilistic timed automaton which substitutes for the nondeterminism of an ordinary timed automaton a new one directly drawn from markov decision processes by means of actions which provide a probabilistic distribution over transitions using buchi acceptance conditions timed automata can refer timing properties as during every open time interval of length 1 at least one message is delivered a policy is a mechanism which solves the non determinism by choosing for each finite run an action and the time moment of the next transition step implied by this action we prove that given a probabilistic timed automaton a there exists a markov memoryless policy which maximizes the probability p of the set of accepting runs realized by this policy this policy as well as the maximal value of p are computable in polytime in the size of the region automaton of a this result provides an algorithm of model checking for properties like there is a policy which realizes a correct behavior of the system with probability at least p

cesaro limits of analytically perturbed stochastic matrices abstract let p 0 be a stochastic matrix with no eigenvalue other than 1 on the unit circle and let p e p 0 a e be an analytic perturbation which is still stochastic for small enough positive e we determine the limit lim e 0 1 n e l 1 n e p l e when n e at various rates

the interacting multiple model algorithm for systems with markovian switching coefficients an important problem in filtering for linear systems with markovian switching coefficients dynamic multiple model systems is the management of hypotheses which is necessary to limit the computational requirements a novel approach to hypotheses merging is presented for this problem the novelty lies in the timing of hypotheses merging when applied to the problem of filtering for a linear system with markovian coefficients the method is an elegant way to derive the interacting multiple model imm algorithm evaluation of the imm algorithm shows that it performs well at a relatively low computational load these results imply a significant change in the state of the art of approximate bayesian filtering for systems with markovian coefficients

on nearly optimal controls of hybrid lqg problems the paper is concerned with hybrid control of a class of linear quadratic gaussian lqg systems modulated by a finite state markov chain it develops approximation schemes for systems involving singularly perturbed markov chains with weak and strong interactions which are useful for natural time scale separation and for large scale markovian systems the presentation encompasses three cases the recurrent markov chains inclusion of transient states and inclusion of absorbing states near optimality of the approximation schemes is obtained and a numerical example is presented computational results indicate our approximation schemes perform well

robust jump linear quadratic control a mode stabilizing solution this correspondence is concerned with optimal feedback control of a class of linear stochastic systems for a quadratic performance index the solution to this problem is known but the resulting optimal regulator does not ensure stability in every mode of operation to improve robustness a new mode stabilizing solution is defined an example illustrates the advantages of the proposed controller

the first laurent series coefficients for singularly perturbed stochastic matrices there are a few procedures for computing the laurent series expansions for the mean passage time matrix and for the deviation matrix of a singularly perturbed markov chain we suggest here a method for computing the first terms in these expansions in a way which highlights the system dynamics in various time scales

maximum principle for optimal control problems of forward backward regime switching system and applications abstract in this paper we derive the stochastic maximum principle for optimal control problems of the forward backward markovian regime switching system the control system is described by forward backward sdes and modulated by continuous time finite state markov chains we first obtain the necessary and sufficient conditions for the optimal control thereafter we apply the maximum principle to recursive utility investment consumption problems and lq problems with markovian regime switching

hybrid filtering for linear systems with non gaussian disturbances this paper is concerned with hybrid filtering of a non gaussian system coupled by a hidden switching process an optimal control approach is used to derive a finite dimensional recursive filter that is optimal in the sense of the most probable trajectory mpt estimate an approximate scheme is obtained when the probability distribution of the switching process converges to a limiting distribution which is often easier to obtain numerical examples are considered and computational experiments are reported in these examples our schemes outperform popular filtering schemes available in the literature

control of dynamic systems under the influence of singularly perturbed markov chains abstract this work is concerned with nearly optimal controls of nonlinear dynamic systems under the influence of singularly perturbed markov chains the underlying markov chains have fast and slow components and their states can be divided into a number of groups within each group of states the chain varies in a fast pace whereas the jumps from one group to another occur relative infrequently to obtain the desired optimality the states of the chain are naturally aggregated in accordance with the transition rates i e replacing the states in a group by a single state to obtain an average system then the averaged system is used as a reference to develop the nearly optimal control for the actual system via comparison control methods the technique used is the method of weak convergence together with the utilization of relaxed control representation

on explicit tamed milstein type scheme for stochastic differential equation with markovian switching abstract we propose a new tamed milstein type scheme for stochastic differential equation with markovian switching when drift coefficient is assumed to grow super linearly the strong rate of convergence is shown to be equal to 1 0 under mild regularity e g once differentiability requirements on drift and diffusion coefficients novel techniques are developed to tackle two fold difficulties arising due to jumps of the markov chain and the reduction of regularity requirements on the coefficients

a two factor stochastic production model with two time scales a two factor production model where one factor is characterized by a continuous state variable and corresponds to the fast mode via a controlled diffusion process whereas the second factor is characterized by a discrete variable and corresponds to the slow mode via a controlled jump process

nonlinear filtering and control of a switching diffusion with small observation noise this paper is concerned with nonlinear filtering and control of a switching diffusion coupled by an unknown markov chain two statistical estimation methods are used to track the unknown markov chain computable approximate filters are obtained based on these methods the filters are then used to construct controls for the partially observed system these controls are shown to be asymptotically optimal as the observation noise tends to zero finally an example is considered and numerical experiments are reported

piecewise monotone filtering in discrete time with small observation noise a discrete time model for filtering with small observation noise is considered a piecewise linear observation function is considered with two intervals of monotonicity a sequential quadratic variation test is found to detect intervals of linearity of the observation function diffusion approximations to certain discrete processes are made to estimate the mean times for reaching a decision and the error probabilities

aggregation of graph models and markov chains by deterministic annealing we consider the problem of simplifying large weighted directed graphs by aggregating nodes and edges this problem is recast as a clustering resource allocation problem and a solution method that incorporates features of the deterministic annealing da algorithm is proposed the novelty in our method is a quantitive measure of dissimilarity that allows us to compare directed graphs of possibly different sizes i e the original and the aggregated graphs the approach we propose is insensitive to initial conditions and less likely to converge to poor local minima than lloyd type algorithms we apply our graph aggregation clustering method to markov chains where low order markov chains that approximate high order chains are obtained through appropriate aggregation of state transition matrices we further develop a decentralized computational scheme to improve tractability of the algorithm

singular perturbations and time scale methods in control theory survey 1976 1983 recent progress in the use of singular perturbation and two time scale methods of modeling and design for control systems is reviewed over 350 references are organized into major problem areas representative issues and results are discussed with a view to outlining research directions and indicating potential areas of application the survey is aimed at engineers and applied mathematicians interested in model order reduction separation of time scales and allied simplified methods of control system analysis and design the exposition does not assume prior knowledge of singular perturbation methods

approximate nonlinear filtering for a two dimensional diffusion with one dimensional observations in a low noise channel the asymptotic behavior of a nonlinear continuous time filtering problem is studied when the variance of the observation noise tends to 0 we suppose that the signal is a two dimensional process from which only one of the components is noisy and that a one dimensional function of this signal depending only on the unnoisy component is observed in a low noise channel an approximate filter is considered in order to solve this problem under some detectability assumptions we prove that the filtering error converges to 0 and an upper bound for the convergence rate is given the efficiency of the approximate filter is compared with the efficiency of the optimal filter and the order of magnitude of the error between the two filters as the observation noise vanishes is obtained

reduced complexity estimation for large scale hidden markov models in this paper we address the problem of reduced complexity estimation of general large scale hidden markov models hmms with underlying nearly completely decomposable discrete time markov chains and finite state outputs an algorithm is presented that computes o spl epsi where spl epsi is the related weak coupling parameter approximations to the aggregate and full order filtered estimates with substantial computational savings these savings are shown to be quite large when the chains have blocks with small individual dimensions some simulation studies are presented to demonstrate the performance of the algorithm

percolation thresholds on planar euclidean relative neighborhood graphs in the present article statistical properties regarding the topology and standard percolation on relative neighborhood graphs rngs for planar sets of points considering the euclidean metric are put under scrutiny rngs belong to the family of proximity graphs i e their edgeset encodes proximity information regarding the close neighbors for the terminal nodes of a given edge therefore they are e g discussed in the context of the construction of backbones for wireless ad hoc networks that guarantee connectedness of all underlying nodes here by means of numerical simulations we determine the asymptotic degree and diameter of rngs and we estimate their bond and site percolation thresholds which were previously conjectured to be nontrivial we compare the results to regular 2d graphs for which the degree is close to that of the rng finally we deduce the common percolation critical exponents from the rng data to verify that the associated universality class is that of standard 2d percolation

constraint markov chains notions of specification implementation satisfaction and refinement together with operators supporting stepwise design constitute a specification theory we construct such a theory for markov chains mcs employing a new abstraction of a constraint mc constraint mcs permit rich constraints on probability distributions and thus generalize prior abstractions such as interval mcs linear polynomial constraints suffice for closure under conjunction respectively parallel composition this is the first specification theory for mcs with such closure properties we discuss its relation to simpler operators for known languages such as probabilistic process algebra despite the generality all operators and relations are computable

two time scale controlled markov chains a decomposition and parallel processing approach this correspondence deals with a class of ergodic control problems for systems described by markov chains with strong and weak interactions these systems are composed of a set of subchains that are weakly coupled using results already available in the literature one formulates a limit control problem the solution of which can be obtained via an associated nondifferentiable convex programming ndcp problem the technique used to solve the ndcp problem is the analytic center cutting plane method accpm which implements a dialogue between on one hand a master program computing the analytical center of a localization set containing the solution and on the other hand an oracle proposing cutting planes that reduce the size of the localization set at each main iteration the interesting aspect of this implementation comes from two characteristics i the oracle proposes cutting planes by solving reduced sized markov decision problems mdp via a linear program lp or a policy iteration method ii several cutting planes can be proposed simultaneously through a parallel implementation on processors the correspondence concentrates on these two aspects and shows on a large scale mdp obtained from the numerical approximation ldquoa la kushner dupuisrdquo of a singularly perturbed hybrid stochastic control problem the important computational speed up obtained

paper on coherency based decomposition algorithms a new algorithm of coherency based decomposition is presented and applied contrary to avramovic and coworkers algorithm it does not necessarily imply in practice the knowledge of the eigenvalues and eigenvectors of the system or an a priori knowledge of the number of coherent areas as long as two classes of dynamics slow and fast exist its applicability to the case of a 48 machine system is demonstrated

weak connections time scales and aggregation of nonlinear systems if nonlinear subsystems with continuous equilibria are weakly connected their local behavior is fast compared with the system wide behavior caused by the connections the slow behavior is described by an aggregate model which appears as a slow subsystem in the singular perturbation form of the model in this way earlier linear aggregation results by simon et al in economics and slow coherency results in power systems are extended to nonlinear systems and related to singular perturbations

two time scale approximation for wonham filters this paper is concerned with approximation of wonham filters a focal point is that the underlying hidden markov chain has a large state space to reduce computational complexity a two time scale approach is developed under time scale separation the state space of the underlying markov chain is divided into a number of groups such that the chain jumps rapidly within each group and switches occasionally from one group to another such structure gives rise to a limit wonham filter that preserves the main features of the filtering process but has a much smaller dimension and therefore is easier to compute using the limit filter enables us to develop efficient approximations and useful filters for hidden markov chains the main advantage of our approach is the reduction of dimensionality

singularly perturbed discrete time markov chains originating from a wide range of applications in optimization and control of large scale systems such as telecommunications queueing networks and manufacturing systems this work is devoted to a class of singularly perturbed discrete time markov chains the states of the markov chain are naturally decomposable into recurrent and transient classes such that within each class the interactions are strong and among different classes the interactions are weak our study focuses on the difference equations representing the probabilityvector and aims at deriving matched asymptotic expansions of the solutions justification and error analysis are also provided both time homogeneous and time inhomogeneous models are examined

a real schur form method for modeling singularly perturbed systems a method is presented for modeling a two time scale system in the singularly perturbed form the method uses an ordered real schur decomposition which can be efficiently computed using standard subroutines from eispack three results are given first it is shown that any two time scale system can be modeled in the singularly perturbed form by a transformation into an ordered real schur form followed by balancing second under some conditions on the ordered real schur decomposition a procedure is given to achieve the modeling task with all fast variables chosen from the original state variables third necessary and sufficient conditions are given to achieve modeling by permutation of the original state variables

stability of hybrid dynamic systems containing singularly perturbed random processes to meet the challenge of handling dynamic systems in which continuous dynamics and discrete events coexist we develop stability analysis of such systems governed by ordinary differential equations and stochastic differential equations with regime switching modulated by a markov chain involving a small parameter the small parameter is used to reflect either the in herent two time scales in the original system or the different rates of regime switching among a large number of states of the discrete events the smaller the parameter is the more rapid switching the system will experience to reduce the complexity one attempts to effectively replace the actual systems by a limit system with simpler structure to ensure the validity of such a replacement in a longtime horizon it is crucial that the original system is stable the fast regime changes and the large state space of the markov chain make the stability analysis difficult under suitable conditions using the limit dynamic systems and the perturbed lyapunov func tion methods we show that if the limit systems are stable then so are the original systems this justifies the replacement of a complicated original system by its limit from a longtime behavior point of view index terms hybrid system lyapunov method markov chain singular perturbation stability switching diffusion

singularly perturbed markov chains with two small parameters a matched asymptotic expansion this work is concerned with asymptotic properties of solutions to forward equations for singularly perturbed markov chains with two small parameters it is motivated by the model of a cost minimizing firm involving production planning and capacity expansion and a two level hierarchical decomposition our effort focuses on obtaining asymptotic expansions of the solutions to the forward equation different from previous work on singularly perturbed markov chains the inner expansion terms are constructed by solving certain partial differential equations the methods of undetermined coefficients are used the error bound is obtained

a unified perturbation analysis framework for countable markov chains abstract in this paper we are devoted to singular perturbation analysis for discrete time or continuous time markov chains we modify and extend the drift condition method well known for regular perturbation to develop a new framework for singular perturbation analysis our results extend and improve the corresponding ones in 2 for singularly perturbed markov chains by allowing a general perturbation form less restrictive conditions and more computable bounds our analysis covers the regular perturbation analysis and hence unifies singular and regular perturbation analysis furthermore our results are illustrated by two two dimensional markov chains including a discrete time queue and a continuous time level dependent quasi birth death process

complexity reduction in fixed lag smoothing for hidden markov models we investigate approximate smoothing schemes for a class of hidden markov models hmms namely hmms with underlying markov chains that are nearly completely decomposable the objective is to obtain substantial computational savings our algorithm can not only be used to obtain aggregate smoothed estimates but can be used also to obtain systematically approximate full order smoothed estimates with computational savings and rigorous performance guarantees unlike many of the aggregation methods proposed earlier

paper optimal control of markov chains admitting strong and weak interactions this paper is devoted to the study of an optimal control problem for a markov chain with generator b ea where e is a small parameter it is shown that an approximate solution can be calculated by a policy improvement algorithm involving computations relative to an aggregated problem the dimension of which is given by n the number of ergodic sets for the b matrix together with a family of decentralized problems the dimensions of which are given by the number of elements in each ergodic set for the b matrix

percolation on a multifractal scale free planar stochastic lattice and its universality class we investigate site percolation on a weighted planar stochastic lattice wpsl which is a multifractal and whose dual is a scale free network percolation is typically characterized by a threshold value p c at which a transition occurs and by a set of critical exponents which describe the critical behavior of the percolation probability p p mean cluster size s p and the correlation length besides the exponent characterizes the cluster size distribution function n s p c and the fractal dimension d f characterizes the spanning cluster we numerically obtain the value of p c and of all the exponents these results suggest that the percolation on wpsl belong to a separate universality class than on all other planar lattices

asymptotically optimal controls of hybrid linear quadratic regulators in discrete time this work develops asymptotically optimal controls for a class of discrete time hybrid systems involving singularly perturbed markov chains having weak and strong interactions the state space of the underlying markov chain is decomposed into a number of recurrent classes and a group of transient states using a hierarchical control approach by aggregating the states in each recurrent class into a single state a continuous time quadratic limit control problem in which the resulting limit markov chain has much smaller state space is derived using the optimal control for the limit problem a control for the original problem is constructed which is shown to be nearly optimal finally a numerical example is given to demonstrate the effectiveness of the approximation scheme

laurent expansion of the inverse of perturbed singular matrices in this paper we describe a numerical algorithm to compute the laurent expansion of the inverse of singularly perturbed matrices the algorithm is based on the resolvent formalism used in complex analysis to study the spectrum of matrices the input of the algorithm are the matrix coefficients of the power series expansion of the perturbed matrix the matrix coefficients of the laurent expansion of the inverse are computed using recursive analytical formulae we show that the computational complexity of the proposed algorithm grows algebraically with the size of the matrix but exponentially with the order of the singularity we apply this algorithm to several matrices that arise in applications we make special emphasis to interpolation problems with radial basis functions

asymptotic optimization of a nonlinear hybrid system governed by a markov decision process we consider in this paper a continuous time stochastic hybrid control system with finite time horizon the objective is to minimize a nonlinear function of the state trajectory the state evolves according to a nonlinear dynamics the parameters of the dynamics of the system may change at discrete times l epsilon l 0 1 according to a controlled markov chain which has finite state and action spaces under the assumption that epsilon is a small parameter we justify an averaging procedure allowing us to establish that our problem can be approximated by the solution of some deterministic optimal control problem

numerical approximation of invariant measures for hybrid diffusion systems this work is concerned with numerical approximation of hybrid diffusions with regime switching modulated by continuous time finite state markov chains when using the euler maruyama approximation algorithms an important question is suppose the measure of the regime switching diffusions converges to its invariant measure will the sequence of measures of the approximation scheme converges to the same limit this paper provides answers to this question by appropriate interpolations and weak convergence methods it shows that a suitably interpolated sequence resulted from the algorithm converges to the switching diffusion the convergence to the invariant measure of the numerical algorithm is also studied

near optimal controls of random switching lq problems with indefinite control weight costs in this paper we consider hybrid controls for a class of linear quadratic problems with white noise perturbation and markov regime switching where the regime switching is modeled by a continuous time markov chain with a large state space and the control weights are indefinite the use of large state space enables us to take various factors of uncertain environment into consideration yet it creates computational overhead and adds difficulties aiming at reduction of complexity we demonstrate how to construct near optimal controls first in the model we introduce a small parameter to highlight the contrast of the weak and strong interactions and fast and slow motions this results in a two time scale formulation in view of the recent developments on lq problems with indefinite control weights and two time scale markov chains we then establish the convergence of the system of riccati equations associated with the hybrid lq problem based on the optimal feedback control of the limit system obtained using the system of riccati equations we construct controls for the original problem and show that such controls are near optimal a numerical demonstration of a simple system is presented here

a method to calculate steady state distributions of large markov chains by aggregating states this paper develops an efficient iterative algorithm to calculate the steady state distribution of nearly all irreducible discrete time markov chains computational experiences suggest that for large markovian systems more than 130 states the proposed algorithm can be ten times faster than standard gaussian elimination in finding solutions to an accuracy of 0 1 the proposed algorithm is developed in three stages first we develop a very efficient algorithm for determining steady state distributions of a restricted class of markovian systems a second result establishes a relationship between a general irreducible markovian system and a system in the restricted class of markovian systems finally we combine the two results to produce an efficient iterative algorithm to solve markov systems the paper concludes with a discussion of the observed performance of the algorithm

markov chains with rare transitions and simulated annealing we consider approximate stationary markov chains in which the entries of the one step transition probability matrix are known to be of different orders of magnitude and whose structure that is the orders of magnitude of the transition probabilities does not change with time for such markov chains we present a method for generating order of magnitude estimates for the t step transition probabilities for any t we then notice that algorithms of the simulated annealing type may be represented by markov chains which are approximately stationary over fairly long time intervals using our results we obtain a characterization of the convergent cooling schedules for the most general class of algorithms of the simulated annealing type

a stochastic multiscale model for electricity generation capacity expansion abstract long term planning for electric power systems or capacity expansion has traditionally been modeled using simplified models or heuristics to approximate the short term dynamics however current trends such as increasing penetration of intermittent renewable generation and increased demand response requires a coupling of both the long and short term dynamics we present an efficient method for coupling multiple temporal scales using the framework of singular perturbation theory for the control of markov processes in continuous time we show that the uncertainties that exist in many energy planning problems in particular load demand uncertainty and uncertainties in generation availability can be captured with a multiscale model we then use a dimensionality reduction technique which is valid if the scale separation present in the model is large enough to derive a computationally tractable model we show that both wind data and electricity demand data do exhibit sufficient scale separation a numerical example using real data and a finite difference approximation of the hamilton jacobi bellman equation is used to illustrate the proposed method we compare the results of our approximate model with those of the exact model we also show that the proposed approximation outperforms a commonly used heuristic used in capacity expansion models

perturbation and stability theory for markov control problems a unified approach to the asymptotic analysis of a markov decision process disturbed by an epsilon additive perturbation is proposed irrespective of whether the perturbation is regular or singular the underlying control problem that needs to be understood is the limit markov control problem the properties of this problem are studied

finite horizon robust state estimation for uncertain finite alphabet hidden markov models with conditional relative entropy constraints we consider a robust state estimation problem for time varying uncertain discrete time homogeneous first order finite state finite alphabet hidden markov models hmms a class of time varying uncertain hmms is considered in which the uncertainty is sequentially described by a conditional relative entropy constraint on perturbed conditional probability measures given a realized observation sequence for this class of uncertain hmms the robust state estimation problem is formulated as a constrained optimization problem using a lagrange multiplier technique and a variational formula for conditional relative entropy the above problem is converted into an unconstrained optimization problem and a problem related to partial information risk sensitive filtering a measure transformation technique and an information state method are employed to solve this equivalent problem related to risk sensitive filtering a characterization of the solution to the robust state estimation problem is also presented

piecewise monotone filtering with small observation noise nonlinear filtering of markov diffusion processes is considered in the case in which a piecewise monotone function of the state is observed with additive small observation noise under a certain detectability hypothesis statistical tests are given to discriminate among the intervals of monotonicity during time intervals in which the state does not cross critical points of the observation function during such time intervals accurate approximate finite dimensional filters can be used

closed form solutions for perpetual american put options with regime switching this paper studies an optimal stopping time problem for pricing perpetual american put options in a regime switching model an explicit optimal stopping rule and the corresponding value function in a closed form are obtained using the modified smooth fit technique the solution is then compared with the numerical results obtained via a dynamic programming approach and also with a two point boundary value differential equation tpbvde method

optimal adaptive lqg control for systems with finite state process parameters the situation where a totally observed process y t is generated by a stochastic differential equation whose parameters evolve on a finite set 1 n according to a stochastic differential equation is considered the optimal control law is sought with respect to quadratic loss functions on y t and the control u t the auxiliary p d e technique of hijab 6 is used together with a nonlinear filter to obtain the solution whose existence depends upon that of a smooth solution to the auxiliary p d e and strong solutions to the system s d e under the given control inputs

controllability stabilizability and continuous time markovian jump linear quadratic control consideration is given to the control of continuous time linear systems that possess randomly jumping parameters which can be described by finite state markov processes the relationship between appropriately defined controllability stabilizability properties and the solution of the infinite time jump linear quadratic jlq optimal control problems is also examined although the solution of the continuous time markov jlq problem with finite or infinite time horizons is known only sufficient conditions for the existence of finite cost constant stabilizing controls for the infinite time problem appear in the literature in this paper necessary and sufficient conditions are established these conditions are based on new definitions of controllability observability stabilizability and detectability that are appropriate for continuous time markovian jump linear systems these definitions play the same role for the jlq problem as the deterministic properties do for the linear quadratic regulator lqr problem

the adaptive lqg problem part i this paper is concerned with a rigorous study of the dual control problem of fel dbaum i e the lqg optimal control problem in the presence of bayesian parameter uncertainty the solution of this problem involves two parts one relating to filtering and one to control although we establish our filtering result in complete generality in the last section most of the paper concentrates on the finite parameter case to ease the exposition the control result that we establish is incomplete in the sense that the smoothness of the optimal cost function is assumed rather than proved nevertheless our results and methods are such that we arrive at a new proof of the classical separation theorem showing that the well known lqg feedback law is optimal within the widest possible class of admissible controls as this new proof avoids all talk of dependence of the sigma algebra on the control weak solutions measure transformation techniques etc we feel that this result will help to clarify what is involved in the classical separation theorem

percolation thresholds on two dimensional voronoi networks and delaunay triangulations the site percolation threshold for the random voronoi network is determined numerically with the result pc 0 714 10 0 000 02 using monte carlo simulation on periodic systems of up to 40 000 sites the result is very close to the recent theoretical estimate pc approximately 0 7151 of neher for the bond threshold on the voronoi network we find pc 0 666 931 0 000 005 implying that for its dual the delaunay triangulation pc 0 333 069 0 000 005 these results rule out the conjecture by hsu and huang that the bond thresholds are 2 3 and 1 3 respectively but support the conjecture of wierman that for fully triangulated lattices other than the regular triangular lattice the bond threshold is less than 2 sin pi 18 approximately 0 3473

the reduction of perturbed markov generators an algorithm exposing the role of transient states a new algorithm for the hierarchical aggregation of singularly perturbed finite state markov processes is derived the approach taken bridges the gap between conceptually simple results for a relatively restricted class of processes and the significantly more complex results for the general case the critical role played by almost transient states is exposed resulting in a straightforward algorithm for the construction of a sequence of aggregate generators associated with various time scales these generators together provide a uniform asymptotic approximation of the original probability transition function

applications of singular perturbation techniques to control problems this paper discusses typical applications of singular perturbation techniques to control problems in the last fifteen years the first three sections are devoted to the standard model and its time scale stability and controllability properties the next two sections deal with linear quadratic optimal control and one with cheap near singular control then the composite control and trajectory optimization are considered in two sections and stochastic control in one section the last section returns to the problem of modeling this time in the context of large scale systems the bibliography contains more than 250 titles

a unified view of aggregation and coherency in networks and markov chains abstract this paper presents a unified treatment of aggregability lumpability coherency reversibility partial balance and similar properties appearing in different fields including power systems and queueing theory a coherency condition well known in power systems implies the existence of a finite state filter for markov chains while aggregability and coherency yield a new condition for decentralized computation of the invariant measure

asymptotic expansions of solutions of integro differential equations for transition densities of singularly perturbed switching diffusions rapid switchings this work develops asymptotic expansions for solutions of integro differential equations arising from transition densities of singularly perturbed switching diffusion processes the underlying processes have two components namely the switching component and the diffusion component processes with rapid switchings are examined here under suitable conditions it is proved that the asymptotic expansion is uniformly valid in the domain of interest

markov decision processes with slow scale periodic decisions we consider a class of discrete time dynamic decision making models which we refer to as periodically time inhomogeneous markov decision processes ptmdps in these models the decision making horizon can be partitioned into intervals calledslow scale cycles ofn 1 epochs the transition law and reward function are time homogeneous over the firstn epochs of each slow scale cycle but distinct at the final epoch the motivation for such models is in applications where decisions of different nature are taken at different time scales i e many low level decisions are made between less frequent high level ones for the ptmdp model we consider the problem of optimizing the expected discounted reward when rewards devalue by a discount factor at the beginning of each slow scale cycle whenn is large initially stationary policies i s p s are natural candidates for optimal policies similar to turnpike policies an initially stationary policy uses the same decision rule for some large number of epochs in each slow scale cycle followed by a relatively short planning horizon of time varying decision rules in this paper we characterize the form of the optimal value as a function ofn establish conditions ensuring the existence of near optimal i s p s and characterize their structure our analysis deals separately with the cases where the time homogeneous part of the system has state dependent and state independent optimal average reward as we illustrate the results in these two distinct cases are qualitatively different

jump linear quadratic gaussian control in continuous time the optimal quadratic control of continuous time linear systems that possess randomly jumping parameters which can be described by finite state markov processes is addressed the systems are also subject to gaussian input and measurement noise the optimal solution for the jump linear quadratic gaussian jlqc problem is given this solution is based on a separation theorem the optimal state estimator is sample path dependent if the plant parameters are constant in each value of the underlying jumping process then the controller portion of the compensator converges to a time invariant control law however the filter portion of the optimal infinite time horizon jlqc compensator is not time invariant thus a suboptimal filter which does converge to a steady state solution under certain conditions is derived and a time invariant compensator is obtained

a reduction process for perturbed markov chains given a perturbed markov chain with generator a varepsilon a 0 varepsilon a 1 varepsilon 2 a 2 cdots it is shown that to each time scale t t varepsilon t varepsilon 2 cdots is associated a reduced markov chain this process of reduction has only finitely many steps it is given in a recursive way and each reduced chain admits as state space the recurrent classes of the reduced chain defined at the previous stage it is shown that if lambda is an eigenvalue different from zero of the generator of the kth reduced chain with multiplicity p then a varepsilon admits p eigenvalues lambda varepsilon such that lambda varepsilon varepsilon k lambda o varepsilon k a basis of ker a 0 resp ker a 0 which is robust to the perturbation is constructed

discrete regulators with time scale separation a procedure for decomposition and control design of linear discrete regulators with fast and slow modes is presented the composite and reduced order controls are formulated in a standard form and upper bounds on their performance are derived an eighth order power model is worked out to demonstrate the analytical development

perturbation of null spaces with application to the eigenvalue problem and generalized inverses we consider properties of a null space of an analytically perturbed matrix in particular we obtain taylor expansions for the eigenvectors which constitute a basis for the perturbed null space furthermore we apply these results to the calculation of puiseux expansion of the perturbed eigenvectors in the case of general eigenvalue problem as well as to the calculation of laurent series expansions for the perturbed group inverse and pseudoinverse matrices

algorithms for singularly perturbed limiting average markov control problems a singularly perturbed markov decision process with the limiting average reward criterion is considered it is assumed that the underlying process is composed of n separate irreducible processes and that the small perturbation is such that it unites these processes into a single irreducible process two algorithms for the solution of the underlying limit markov control problem are presented the first of these is a linear program possessing the wolfe dantzig structure inherited from the ergodic nearly decomposable assumption in the model the second is an aggregation disaggregation policy improvement algorithm

optimal kullback leibler aggregation via spectral theory of markov chains this paper is concerned with model reduction for complex markov chain models the kullback leibler divergence rate is employed as a metric to measure the difference between the markov model and its approximation for a certain relaxation of the bi partition model reduction problem the solution is shown to be characterized by an associated eigenvalue problem the form of the eigenvalue problem is closely related to the markov spectral theory for model reduction this result is the basis of a heuristic proposed for the m ary partition problem resulting in a practical recursive algorithm the results are illustrated with examples

a singular perturbation approach to modeling and control of markov chains finite state continuous time markov processes with weak interactions are modeled as singularly perturbed systems aggregate states are obtained using a grouping algorithm two time scale expansions simplify cost equations and lead to decentralized optimization algorithms

minimal counterexamples for linear time probabilistic verification abstract counterexamples for property violations have a number of important applications like supporting the debugging of erroneous systems and verifying large systems via counterexample guided abstraction refinement in this paper we propose the usage of minimal critical subsystems of discrete time markov chains and markov decision processes as counterexamples for violated regular properties minimality can thereby be defined in terms of the number of states or transitions this problem is known to be np complete for markov decision processes we show how to compute such subsystems using mixed integer linear programming and evaluate the practical applicability in a number of experiments they show that our method yields substantially smaller counterexample than using existing techniques

stochastic dynamic job shops and hierarchical production planning this paper presents an asymptotic analysis of hierarchical production planning in a general manufacturing system consisting of a network of unreliable machines producing a variety of products the concept of a dynamic job shop is introduced by interpreting the system as a directed graph and the structure of the system dynamics is characterized for its use in the asymptotic analysis the optimal control problem for the system is a state constrained problem since the number of parts in any buffer between any two machines must remain nonnegative a limiting problem is introduced in which the stochastic machine capacities are replaced by corresponding equilibrium mean capacities as the rate of change in machine states approaches infinity the value function of the original problem is shown to converge to that of the limiting problem and the convergence rate is obtained furthermore near optimal controls for the original problem are constructed from near optimal controls of the limiting problem and an error estimate is obtained on the near optimality of the constructed controls

bsdes with regime switching weak convergence and applications abstract this paper is concerned with a system of backward stochastic differential equations bsdes with regime switching the bsdes are coupled by a finite state markov chain the underlying markov chain is assumed to have a two time scale or weak and strong interactions structure namely the states of the markov chain can be divided into a number of groups so that the chain jumps rapidly within a group and slowly between the groups it is shown in this paper that the original bsde system can be approximated by a limit system in which the states in each group are aggregated out and replaced by a single state in particular it is proved that the solution of the original bsde system converges weakly under the meyer zheng topology as the fast jump rate goes to infinity the limit process is a solution of aggregated bsdes which can be determined by the corresponding martingale problem the results are applied to a set of partial differential equations and used to validate their convergence to the corresponding limit system finally a numerical example is given to demonstrate the approximation results

large population cost coupled lqg problems with nonuniform agents individual mass behavior and decentralized varepsilon nash equilibria we consider linear quadratic gaussian lqg games in large population systems where the agents evolve according to nonuniform dynamics and are coupled via their individual costs a state aggregation technique is developed to obtain a set of decentralized control laws for the individuals which possesses an epsiv nash equilibrium property a stability property of the mass behavior is established and the effect of inaccurate population statistics on an isolated agent is also analyzed by variational techniques

symbolic optimal expected time reachability computation and controller synthesis for probabilistic timed automata in this paper we consider the problem of computing the optimal minimum or maximum expected time to reach a target and the synthesis of an optimal controller for a probabilistic timed automaton pta although this problem admits solutions that employ the digital clocks abstraction or statistical model checking symbolic methods based on zones and priced zones fail due to the difficulty of incorporating probabilistic branching in the context of dense time we work in a generalisation of the setting introduced by asarin and maler for the corresponding problem for timed automata where simple and nice functions are introduced to ensure finiteness of the dense time representation we find restrictions sufficient for value iteration to converge to the optimal expected time on the uncountable markov decision process representing the semantics of a pta we formulate bellman operators on the backwards zone graph of a pta and prove that value iteration using these operators equals that computed over the pta s semantics this enables us to extract an optimal controller from value iteration in the standard way