rotators periodicity and absence of diffusion in cyclic cellular automata cyclic cellular automata are two dimensional cellular automata which generalize lattice versions of the lorentz gas and certain biochemistry models of artificial life we show that rotators and time reversibility play a special role in the creation of closed orbits in cyclic cellular automata we also prove that almost every orbit is closed periodic and the absence of diffusion for the flipping rotator model also known as the ant

computationally feasible bounds for partially observed markov decision processes a partially observed markov decision process pomdp is a sequential decision problem where information concerning parameters of interest is incomplete and possible actions include sampling surveying or otherwise collecting additional information such problems can theoretically be solved as dynamic programs but the relevant state space is infinite which inhibits algorithmic solution this paper explains how to approximate the state space by a finite grid of points and use that grid to construct upper and lower value function bounds generate approximate nonstationary and stationary policies and bound the value loss relative to optimal for using these policies in the decision problem a numerical example illustrates the methodology

partially observable markov decision processes incorporating epistemic uncertainties abstract the use of markov decision processes for inspection maintenance and rehabilitation of civil engineering structures relies on the use of several transition matrices related to the stochastic degradation process maintenance actions and imperfect inspections point estimators for these matrices are usually used and they are evaluated using statistical inference methods and or expert evaluation methods thus considerable epistemic uncertainty often veils the true values of these matrices our contribution through this paper is threefold first we present a methodology for incorporating epistemic uncertainties in dynamic programming algorithms used to solve finite horizon markov decision processes which may be partially observable second we propose a methodology based on the use of dirichlet distributions which answers in our sense much of the controversy found in the literature about estimating markov transition matrices third we show how the complexity resulting from the use of monte carlo simulations for the transition matrices can be greatly overcome in the framework of dynamic programming the proposed model is applied to concrete bridge under degradation in order to provide the optimal strategy for inspection and maintenance the influence of epistemic uncertainties on the optimal solution is underlined through sensitivity analysis regarding the input data

algebraic spatial correlations in lattice gas automata violating detailed balance in this paper we discuss the existence of generic long range correlations in spatially homogeneous and stable equilibrium states of closed lattice gas automata whose stochastic collision rules violate the symmetry conditions of detailed balance and in addition satisfy local conservation laws such correlations occur even though the collision rules are strictly local and invariant under all symmetries of the lattice first a phenomenological langevin equation approach is discussed next we present a theoretical analysis on the basis of an approximate microscopic ring kinetic theory this theory is used to calculate the amplitude ofr tails in the spatial correlations and the result is compared with computer simulations

robust mdps with k rectangular uncertainty markov decision processes are a common tool for modeling sequential planning problems under uncertainty in almost all realistic situations the system model cannot be perfectly known and must be approximated or estimated thus we consider markov decision processes under parameter uncertainty which effectively adds a second layer of uncertainty most previous studies restrict to the case that uncertainties among different states are uncoupled which leads to conservative solutions on the other hand robust mdps with general coupled uncertainty sets are known to be computationally intractable in this paper we make a first attempt at identifying subclasses of coupled uncertainty that are flexible enough to overcome conservativeness yet still lead to tractable problems we propose a new class of uncertainty sets termed k rectangular uncertainty sets a geometric concept defined by the cardinality of possible conditional projections of the uncertainty set the proposed scheme can model several intuitive

reinforcement learning state estimator in this study we propose a novel use of reinforcement learning for estimating hidden variables and parameters of nonlinear dynamical systems a critical issue in hidden state estimation is that we cannot directly observe estimation errors however by defining errors of observable variables as a delayed penalty we can apply a reinforcement learning frame work to state estimation problems specifically we derive a method to construct a nonlinear state estimator by finding an appropriate feedback input gain using the policy gradient method we tested the proposed method on single pendulum dynamics and show that the joint angle variable could be successfully estimated by observing only the angular velocity and vice versa in addition we show that we could acquire a state estimator for the pendulum swing up task in which a swing up controller is also acquired by reinforcement learning simultaneously furthermore we demonstrate that it is possible to estimate the dynamics of the pendulum itself while the hidden variables are estimated in the pendulum swing up task application of the proposed method to a two linked biped model is also presented

latent structure analysis of a set of multidimensional contingency tables abstract statistical methods are introduced for latent structure analysis of a set of two or more multidimensional contingency tables three basic classes of models are considered a models that assume complete homogeneity across tables b models that allow partial homogeneity across tables and c models that allow complete heterogeneity methods are presented for testing whether these models are congruent with the data in the tables and for assessing the significance of differences among the tables in the estimated parameters to illustrate the wide applicability of these models and methods we present analyses of two quite different sets of data

discounted robust stochastic games and an application to queueing control this paper presents a robust optimization model for n person finite state action stochastic games with incomplete information we consider nonzero sum discounted stochastic games in which none of the players knows the true data of a game and each player adopts a robust optimization approach to address the uncertainty we call these games discounted robust stochastic games such games allow us to use simple uncertainty sets for the unknown data and eliminate the need to have an a priori probability distribution over a set of games we prove the existence of equilibrium points and propose an explicit mathematical programming formulation for an equilibrium calculation we illustrate the use of discounted robust stochastic games in a single server queueing control problem

real time dynamic programming for markov decision processes with imprecise probabilities markov decision processes have become the standard model for probabilistic planning however when applied to many practical problems the estimates of transition probabilities are inaccurate this may be due to conflicting elicitations from experts or insufficient state transition information the markov decision process with imprecise transition probabilities mdp ips was introduced to obtain a robust policy where there is uncertainty in the transition although it has been proposed a symbolic dynamic programming algorithm for mdp ips called spudd ip that can solve problems up to 22 state variables in practice solving mdp ip problems is time consuming in this paper we propose efficient algorithms for a more general class of mdp ips called stochastic shortest path mdp ips ssp mdp ips that use initial state information to solve complex problems by focusing on reachable states the l rtdp ip algorithm a labeled real time dynamic programming algorithm for ssp mdp ips is proposed together with three different methods for sampling the next state it is shown here that the convergence of l rtdp ip can be obtained by using any of these three methods although the bellman backups for this class of problems prescribe a minimax optimization as far as we are aware this is the first asynchronous algorithm for ssp mdp ips given in terms of a general set of probability constraints that requires non linear optimization over imprecise probabilities in the bellman backup our results show up to three orders of magnitude speedup for l rtdp ip when compared with the spudd ip algorithm

reachability analysis of uncertain systems using bounded parameter markov decision processes verification of reachability properties for probabilistic systems is usually based on variants of markov processes current methods assume an exact model of the dynamic behavior and are not suitable for realistic systems that operate in the presence of uncertainty and variability this research note extends existing methods for bounded parameter markov decision processes bmdps to solve the reachability problem bmdps are a generalization of mdps that allows modeling uncertainty our results show that interval value iteration converges in the case of an undiscounted reward criterion that is required to formulate the problems of maximizing the probability of reaching a set of desirable states or minimizing the probability of reaching an unsafe set analysis of the computational complexity is also presented

low viscosity lattice gases a class of lattice gas models are studied which are variants of the fchc model the aim is to achieve the highest possible reynolds coefficient inverse dimensionless viscosity for efficient simulations of the three dimensional incompressible navier stokes equations the models include an arbitrary number of rest particles and violation of semi detailed balance within the framework of the boltzmann approximation exact expressions are obtained for the reynolds coefficients the minimization of the viscosity is done by solving a hitchcock type optimization problem for the fine tuning of the collision rules when the number of rest particles exceeds one there is a range of densities at which the viscosity takes negative values various optimal models with up to 26 bits per node have been implemented on a cray 2 and their true transport coefficients have been measured with good accuracy fairly large discrepancies with boltzmann values are observed when semi detailed balance is violated in particular no negative viscosity is obtained still the best model has a reynolds coefficient of 13 5 twice that of the best previously implemented model and thus is about 16 times more efficient computationally suggestions are made for further improvements it is proposed to use models with very high reynolds coefficients for sub grid scale modeling of turbulent flows

markov chains for exploring posterior distributions several markov chain methods are available for sampling from a posterior distribution two important examples are the gibbs sampler and the metropolis algorithm in addition several strategies are available for constructing hybrid algorithms this paper outlines some of the basic methods and strategies and discusses some related theoretical and practical issues on the theoretical side results from the theory of general state space markov chains can be used to obtain convergence rates laws of large numbers and central limit theorems for estimates obtained from markov chain methods these theoretical results can be used to guide the construction of more efficient algorithms for the practical use of markov chain methods standard simulation methodology provides several variance reduction techniques and also give guidance on the choice of sample size and allocation

generalized boltzmann equation for lattice gas automata in this paper a theory is formulated that predicts velocity and spatial correlations between occupation numbers that occur in lattice gas automata violating semi detailed balance starting from a coupled bbgky hierarchy for then particle distribution functions cluster expansion techniques are used to derive approximate kinetic equations in zeroth approximation the standard nonlnear boltzmann equation is obtained the next approximation yields the ring kinetic equation similar to that for hard sphere systems describing the time evolution of pair correlations the ring equation is solved to determine the nonvanishing pair correlation functions in equilibrium for two models that violate semidetailed balance one is a model of interacting random walkers on a line the other one is a two dimensional fluid type model on a triangular lattice the numerical predictions agree very well with computer simulations

reinforcement learning in continuous time and space this article presents a reinforcement learning framework for continuous time dynamical systems without a priori discretization of time state and action based on the hamilton jacobi bellman hjb equation for infinite horizon discounted reward problems we derive algorithms for estimating value functions and improving policies with the use of function approximators the processof value function estimation is formulated asthe minimization of a continuous time form of the temporal difference td error update methods based on backward euler approximation and exponential eligibility traces are derived and their correspondences with the conventional residual gradient td 0 and td lambda algorithms are shown for policy improvement two methods a continuous actor critic method and a value gradient based greedy policy are formulated as a special case of the latter a nonlinear feedback control law using the value gradient and the model of the input gain is derived the advantage updating a model free algorithm derived previously is also formulated in the hjbbased framework the performance of the proposed algorithms is first tested in a nonlinear control task of swinging a pendulum up with limited torque it is shown in the simulations that 1 the task is accomplished by the continuous actor critic method in a number of trials several times fewer than by the conventional discrete actor critic method 2 among the continuous policy update methods the value gradient based policy with a known or learned dynamic model performs several times better than the actor critic method and 3 a value function update using exponential eligibility traces is more efficient and stable than that based on euler approximation the algorithms are then tested in a higher dimensional task cartpole swing up this task is accomplished in several hundred trials using the value gradient based policy with a learned dynamic model

solving factored mdps using non homogeneous partitions we present an algorithm for aggregating states in solving large mdps represented as factored mdps using search by successive refinement in the space of non homogeneous partitions homogeneity is defined in terms of stochastic bisimulation and reward equivalence within blocks of a partition since homogeneous partitions that define equivalent reduced state space mdps can have a large number of blocks we relax the requirement of homogeneity the algorithm constructs approximate aggregate mdps from non homogeneous partitions solves the aggregate mdps exactly and then uses the resulting value functions as part of a heuristic in refining the current best nonhomogeneous partition we outline the theory motivating the use of this heuristic and present empirical results in addition to investigating more exhaustive local search methods we explore the use of techniques derived from research on discretizing continuous state spaces finally we compare the results from our algorithms which search in the space of non homogeneous partitions with exact and approximate algorithms which represent homogeneous and approximately homogeneous partitions as decision trees or algebraic decision diagrams

relation between the lattice boltzmann equation and the navier stokes equations abstract it is shown that the lattice gas boltzmann equation may be rewritten in a form which brings out its close relation with the navier stokes equations various consequences are pointed out

robust control of markov decision processes with uncertain transition matrices optimal solutions to markov decision problems may be very sensitive with respect to the state transition probabilities in many practical problems the estimation of these probabilities is far from accurate hence estimation errors are limiting factors in applying markov decision processes to real world problems we consider a robust control problem for a finite state finite action markov decision process where uncertainty on the transition matrices is described in terms of possibly nonconvex sets we show that perfect duality holds for this problem and that as a consequence it can be solved with a variant of the classical dynamic programming algorithm the robust dynamic programming algorithm we show that a particular choice of the uncertainty sets involving likelihood regions or entropy bounds leads to both a statistically accurate representation of uncertainty and a complexity of the robust recursion that is almost the same as that of the classical recursion hence robustness can be added at practically no extra computing cost we derive similar results for other uncertainty sets including one with a finite number of possible values for the transition matrices we describe in a practical path planning example the benefits of using a robust strategy instead of the classical optimal strategy even if the uncertainty level is only crudely guessed the robust strategy yields a much better worst case expected travel time

value function based reinforcement learning in changing markovian environments the paper investigates the possibility of applying value function based reinforcement learning rl methods in cases when the environment may change over time first theorems are presented which show that the optimal value function of a discounted markov decision process mdp lipschitz continuously depends on the immediate cost function and the transition probability function dependence on the discount factor is also analyzed and shown to be non lipschitz afterwards the concept of e mdps is introduced which is a generalization of mdps and e mdps in this model the environment may change over time more precisely the transition function and the cost function may vary from time to time but the changes must be bounded in the limit then learning algorithms in changing environments are analyzed a general relaxed convergence theorem for stochastic iterative algorithms is presented we also demonstrate the results through three classical rl methods asynchronous value iteration q learning and temporal difference learning finally some numerical experiments concerning changing environments are presented

partially observable markov decision processes with imprecise parameters this study extends the framework of partially observable markov decision processes pomdps to allow their parameters i e the probability values in the state transition functions and the observation functions to be imprecisely specified it is shown that this extension can reduce the computational costs associated with the solution of these problems first the new framework pomdps with imprecise parameters pomdpips is formulated we consider 1 the interval case in which each parameter is imprecisely specified by an interval that indicates possible values of the parameter and 2 the point set case in which each probability distribution is imprecisely specified by a set of possible distributions second a new optimality criterion for pomdpips is introduced as in pomdps the criterion is to regard a policy i e an action selection rule as optimal if it maximizes the expected total reward the expected total reward however cannot be calculated precisely in pomdpips because of the parameter imprecision instead we estimate the total reward by adopting arbitrary second order beliefs i e beliefs in the imprecisely specified state transition functions and observation functions although there are many possible choices for these second order beliefs we regard a policy as optimal as long as there is at least one of such choices with which the policy maximizes the total reward thus there can be multiple optimal policies for a pomdpip we regard these policies as equally optimal and aim at obtaining one of them by appropriately choosing which second order beliefs to use in estimating the total reward computational costs incurred in obtaining such an optimal policy can be reduced significantly we provide an exact solution algorithm for pomdpips that does this efficiently third the performance of such an optimal policy as well as the computational complexity of the algorithm are analyzed theoretically last empirical studies show that our algorithm quickly obtains satisfactory policies to many pomdpips

error bounds for finite step approximations for solving infinite horizon controlled markov set chains this note considers finite step approximations for solving an infinite horizon controlled markov set chain problem with finite state and action spaces we develop a value iteration type algorithm based on the optimality equation developed by kurano et al and analyze an error bound relative to the optimal value that satisfies the optimality equation from the successive approximation we further analyze an error bound of the approximate control policy defined from a finite step approximate value by applying the value iteration type algorithm

online adaptive optimal control for continuous time nonlinear systems with completely unknown dynamics an online adaptive optimal control is proposed for continuous time nonlinear systems with completely unknown dynamics which is achieved by developing a novel identifier critic based approximate dynamic programming algorithm with a dual neural network nn approximation structure first an adaptive nn identifier is designed to obviate the requirement of complete knowledge of system dynamics and a critic nn is employed to approximate the optimal value function then the optimal control law is computed based on the information from the identifier nn and the critic nn so that the actor nn is not needed in particular a novel adaptive law design method with the parameter estimation error is proposed to online update the weights of both identifier nn and critic nn simultaneously which converge to small neighbourhoods around their ideal values the closed loop system stability and the convergence to small vicinity around the optimal solution are all proved by means of the lyapunov theory the proposed ada

integral q learning and explorized policy iteration for adaptive optimal control of continuous time linear systems this paper proposes an integral q learning for continuous time ct linear time invariant lti systems which solves a linear quadratic regulation lqr problem in real time for a given system and a value function without knowledge about the system dynamics a and b here q learning is referred to as a family of reinforcement learning methods which find the optimal policy by interaction with an uncertain environment in the evolution of the algorithm we first develop an explorized policy iteration pi method which is able to deal with known exploration signals then the integral q learning algorithm for ct lti systems is derived based on this pi and the variants of q functions derived from the singular perturbation of the control input the proposed q learning scheme evaluates the current value function and the improved control policy at the same time and are proven stable and convergent to the lq optimal solution provided that the initial policy is stabilizing for the proposed algorithms practical online implementation methods are investigated in terms of persistency of excitation pe and explorations finally simulation results are provided for the better comparison and verification of the performance

sampling based approaches to calculating marginal densities abstract stochastic substitution the gibbs sampler and the sampling importance resampling algorithm can be viewed as three alternative sampling or monte carlo based approaches to the calculation of numerical estimates of marginal probability distributions the three approaches will be reviewed compared and contrasted in relation to various joint probability structures frequently encountered in applications in particular the relevance of the approaches to calculating bayesian posterior densities for a variety of structured models will be discussed and illustrated

latent class model diagnosis summary in many areas of medical research such as psychiatry and gerontology latent class variables are used to classify individuals into disease categories often with the intention of hierarchical modeling problems arise when it is not clear how many disease classes are appropriate creating a need for model selection and diagnostic techniques previous work has shown that the pearson 2 statistic and the log likelihood ratio g2 statistic are not valid test statistics for evaluating latent class models other methods such as information criteria provide decision rules without providing explicit information about where discrepancies occur between a model and the data identifiability issues further complicate these problems this paper develops procedures for assessing markov chain monte carlo convergence and model diagnosis and for selecting the number of categories for the latent variable based on evidence in the data using markov chain monte carlo techniques simulations and a psychiatric example are presented to demonstrate the effective use of these methods

the calculation of posterior distributions by data augmentation abstract the idea of data augmentation arises naturally in missing value problems as exemplified by the standard ways of filling in missing cells in balanced two way tables thus data augmentation refers to a scheme of augmenting the observed data so as to make it more easy to analyze this device is used to great advantage by the em algorithm dempster laird and rubin 1977 in solving maximum likelihood problems in situations when the likelihood cannot be approximated closely by the normal likelihood maximum likelihood estimates and the associated standard errors cannot be relied upon to make valid inferential statements from the bayesian point of view one must now calculate the posterior distribution of parameters of interest if data augmentation can be used in the calculation of the maximum likelihood estimate then in the same cases one ought to be able to use it in the computation of the posterior distribution it is the purpose of this article to explain how this can be done the basic idea

robust planning with incomplete domain models most current planners assume complete domain models and focus on generating correct plans unfortunately domain modeling is a laborious and error prone task thus real world agents have to plan with incomplete domain models while domain experts cannot guarantee completeness often they are able to circumscribe the incompleteness of the model by providing annotations as to which parts of the domain model may be incomplete in this paper we study planning problems with incomplete domain models where the annotations specify possible preconditions and effects of actions we show that the problem of assessing the quality of a plan or its plan robustness is p complete establishing its equivalence with the weighted model counting problems we present two approaches to synthesizing robust plans while the method based on the compilation to conformant probabilistic planning is much intuitive its performance appears to be limited to only small problem instances our second approach based on stochastic heuristic search works well for much larger problems it aims to use the robustness measure directly for estimating heuristic distance which is then used to guide the search our planning system pisa outperforms a state of the art planner handling incomplete domain models in most of the tested domains both in terms of plan quality and planning time finally we also present an extension of pisa called cpisa that is able to exploit the available of past successful plan traces to both improve the robustness of the synthesized plans and reduce the domain modeling burden

approximate robust policy iteration using multilayer perceptron neural networks for discounted infinite horizon markov decision processes with uncertain correlated transition matrices we study finite state finite action discounted infinite horizon markov decision processes with uncertain correlated transition matrices in deterministic policy spaces existing robust dynamic programming methods cannot be extended to solving this class of general problems in this paper based on a robust optimality criterion an approximate robust policy iteration using a multilayer perceptron neural network is proposed it is proven that the proposed algorithm converges in finite iterations and it converges to a stationary optimal or near optimal policy in a probability sense in addition we point out that sometimes even a direct enumeration may not be applicable to addressing this class of problems however a direct enumeration based on our proposed maximum value approximation over the parameter space is a feasible approach we provide further analysis to show that our proposed algorithm is more efficient than such an enumeration method for various scenarios

implementation of the fchc lattice gas model on the connection machine the 4 dimensional fchc lattice gas model has been implemented on a connection machine cm 2 with 16k processors symmetries are used to reduce the collision table to a size that fits into local memory this method avoids the degradation of the reynolds coefficientr but at the price of increased computing time bit shuffling between parallel lattices is introduced to reduce the discrepancy between measured viscosities and those predicted from the boltzmann approximation thereby a model with a negative shear viscosity is obtained a fluid having a uniform initial velocity is unstable and organized nonuniform motions develop because of the buildup of very strong correlations between the parallel lattices the discrepancy with the boltzmann values decreases only very slowly with the number of parallel lattices

h theorem and generalized semi detailed balance condition for lattice gas systems for lattice gas systems obeying fermi dirac statistics an h theorem can be proved with a more general condition that the semi detailed balance condition this new condition allows more flexible transition rates among states so that it has broader applicability for various lattice gas models including those which have multiple phase properties

self diffusion in ca fluids tagged particle properties such as the diffusion coefficients and velocity autocorrelation function are calculated in the mean field approximation for all standard lattice gas cellular automata defined on hyper cubic square and triangular lattices and on a line tagged particle dynamics is introduced through maximally random or minimally random collision rules for a completely filled lattice the former reduces to a random walk the latter to ballistic motion

parameter imprecision in finite state finite action dynamic programs in order to model parameter imprecision associated with a problem s reward or preference structure we examine a finite state finite action dynamic program having a one step transition value function that is affine in an imprecisely known parameter for the finite horizon case we also assume that the terminal value function is affine in the imprecise parameter we assume that the parameter of interest has no dynamics no new information about its value is received once the decision process begins and its imprecision is described by set inclusion we seek the set of all parameter independent strategies that are optimal for some value of the imprecisely known parameter we present a successive approximations procedure for solving the finite horizon case and a policy iteration procedure for determining the solution of the discounted infinite horizon case these algorithms are then applied to a decision analysis problem with imprecise utility function and to a markov decision process with imprecise reward structure we also present conditions that guarantee the existence of a parameter independent strategy that maximizes with respect to all other parameter invariant strategies the minimum value of its expected reward function over all possible parameter values

model based reinforcement learning for approximate optimal regulation reinforcement learning rl based online approximate optimal control methods applied to deterministic systems typically require a restrictive persistence of excitation pe condition for convergence this paper develops a concurrent learning cl based implementation of model based rl to solve approximate optimal regulation problems online under a pe like rank condition the development is based on the observation that given a model of the system rl can be implemented by evaluating the bellman error at any number of desired points in the state space in this result a parametric system model is considered and a cl based parameter identifier is developed to compensate for uncertainty in the parameters uniformly ultimately bounded regulation of the system states to a neighborhood of the origin and convergence of the developed policy to a neighborhood of the optimal policy are established using a lyapunov based analysis and simulation results are presented to demonstrate the performance of the developed controller

efficient solutions to factored mdps with imprecise transition probabilities when modeling real world decision theoretic planning problems in the markov decision process mdp framework it is often impossible to obtain a completely accurate estimate of transition probabilities for example natural uncertainty arises in the transition specification due to elicitation of mdp transition models from an expert or estimation from data or non stationary transition distributions arising from insufficient state knowledge in the interest of obtaining the most robust policy under transition uncertainty the markov decision process with imprecise transition probabilities mdp ips has been introduced to model such scenarios unfortunately while various solution algorithms exist for mdp ips they often require external calls to optimization routines and thus can be extremely time consuming in practice to address this deficiency we introduce the factored mdp ip and propose efficient dynamic programming methods to exploit its structure noting that the key computational bottleneck in the solution of factored mdp ips is the need to repeatedly solve nonlinear constrained optimization problems we show how to target approximation techniques to drastically reduce the computational overhead of the nonlinear solver while producing bounded approximately optimal solutions our results show up to two orders of magnitude speedup in comparison to traditional flat dynamic programming approaches and up to an order of magnitude speedup over the extension of factored mdp approximate value iteration techniques to mdp ips while producing the lowest error of any approximation algorithm evaluated

evolutionary policy iteration for solving markov decision processes we propose a novel algorithm called evolutionary policy iteration epi for solving infinite horizon discounted reward markov decision processes epi inherits the spirit of policy iteration but eliminates the need to maximize over the entire action space in the policy improvement step so it should be most effective for problems with very large action spaces epi iteratively generates a population or a set of policies such that the performance of the elite policy for a population monotonically improves with respect to a defined fitness function epi converges with probability one to a population whose elite policy is an optimal policy epi is naturally parallelizable and along this discussion a distributed variant of pi is also studied

viscous drag by cellular automata a simple method to compute the drag coefficient of two dimensional bodies with arbitrary shapes is presented the procedure is based on cellular automata as an extreme idealization of the molecular dynamics of a viscous fluid we verify the algorithm by examples and obtain results in quantitative agreement with experiments even when eddies behind the obstacle are formed

markov decision processes with imprecise transition probabilities we present new numerical algorithms and bounds for the infinite horizon discrete stage finite state and action markov decision process with imprecise transition probabilities we assume that the transition probability mass vector for each state and action is described by a finite number of linear inequalities this model of imprecision appears to be well suited for describing statistically determined confidence limits and or natural language statements of likelihood the numerical procedures for calculating an optimal max min strategy are based on successive approximations reward revision and modified policy iteration the bounds that are determined are at least as tight as currently available bounds for the case where the transition probabilities are precise

on bayesian analysis of mixtures with an unknown number of components discussion author s reply new methodology for fully bayesian mixture analysis is developed making use of reversible jump markov chain monte carlo methods that are capable of jumping between the parameter subspaces corresponding to different numbers of components in the mixture a sample from the full joint distribution of all unknown variables is thereby generated and this can be used as a basis for a thorough presentation of many aspects of the posterior distribution the methodology is applied here to the analysis of univariate normal mixtures using a hierarchical prior model that offers an approach to dealing with weak prior information while avoiding the mathematical pitfalls of using improper priors in the mixture context

bounded parameter markov decision process in this paper we introduce the notion of a em bounded parameter markov decision process as a generalization of the traditional em exact mdp a bounded parameter mdp is a set of exact mdps specified by giving upper and lower bounds on transition probabilities and rewards all the mdps in the set share the same state and action space bounded parameter mdps can be used to represent variation or uncertainty concerning the parameters of sequential decision problems bounded parameter mdps can also be used in aggregation schemes to represent the variation in the transition probabilities for different base states aggregated together in the same aggregate state we introduce em interval value functions as a natural extension of traditional value functions an interval value function assigns a closed real interval to each state representing the assertion that the value of that state falls within that interval an interval value function can be used to bound the performance of a policy over the set of exact mdps associated with a given bounded parameter mdp we describe an iterative dynamic programming algorithm called em interval policy evaluation which computes an interval value function for a given bounded parameter mdp and specified policy interval policy evaluation on a policy policy computes the most restrictive interval value function that is sound i e that bounds the value function for policy in every exact mdp in the set defined by the bounded parameter mdp a simple modification of interval policy evaluation results in a variant of value iteration bellman57 that we call em interval value iteration which computes a policy for an bounded parameter mdp that is optimal in a well defined sense

a latent transition model with logistic regression latent transition models increasingly include covariates that predict prevalence of latent classes at a given time or transition rates among classes over time in many situations the covariate of interest may be latent this paper describes an approach for handling both manifest and latent covariates in a latent transition model a bayesian approach via markov chain monte carlo mcmc is employed in order to achieve more robust estimates a case example illustrating the model is provided using data on academic beliefs and achievement in a low income sample of adolescents in the united states

relaxation and transport in fchc lattice gases fchc lattice gases are the basic models for studying flow problems in three dimensional systems this paper presents a self contained theoretical analysis and some computer simulations of such lattice gases extended to include an arbitrary number of rest particles with special emphasis on non semi detailed balance nsdb models the special fchc lattice symmetry guarantees isotropy of the navier stokes equations and enumerates the 12 spurious conservation laws staggered momenta the kinetic theory is based on the mean field approximation or the nonlinear boltzmann equation it is shown how calculation of the eigenvalues of the linearized boltzmann equation offers a simple alternative to the chapman enskog method or the multi time scale methods for calculating transport coefficients and relaxation rates the simulated values for the speed of sound in nsdb models slightly disagree with the boltzmann prediction

robust optimality for discounted infinite horizon markov decision processes with uncertain transition matrices we study finite state finite action discounted infinite horizon markov decision processes with uncertain transition matrices in the deterministic policy space the transition matrices are classified as either independent or correlated a generalized robust optimality criterion which can be degenerated to some popular optimality criteria is proposed under which an optimal or near optimal policy exists for any uncertain transition matrix theorems are developed to guarantee a stationary policy being optimal or near optimal in the deterministic policy space

an analysis of temporal difference learning with function approximation we discuss the temporal difference learning algorithm as applied to approximating the cost to go function of an infinite horizon discounted markov chain the algorithm we analyze updates parameters of a linear function approximator online during a single endless trajectory of an irreducible aperiodic markov chain with a finite or infinite state space we present a proof of convergence with probability one a characterization of the limit of convergence and a bound on the resulting approximation error furthermore our analysis is based on a new line of reasoning that provides new intuition about the dynamics of temporal difference learning in addition to proving new and stronger positive results than those previously available we identify the significance of online updating and potential hazards associated with the use of nonlinear function approximators first we prove that divergence may occur when updates are not based on trajectories of the markov chain this fact reconciles positive and negative results that have been discussed in the literature regarding the soundness of temporal difference learning second we present an example illustrating the possibility of divergence when temporal difference learning is used in the presence of a nonlinear function approximator

percentile optimization for markov decision processes with parameter uncertainty markov decision processes are an effective tool in modeling decision making in uncertain dynamic environments because the parameters of these models typically are estimated from data or learned from experience it is not surprising that the actual performance of a chosen strategy often differs significantly from the designer s initial expectations due to unavoidable modeling ambiguity in this paper we present a set of percentile criteria that are conceptually natural and representative of the trade off between optimistic and pessimistic views of the question we study the use of these criteria under different forms of uncertainty for both the rewards and the transitions some forms are shown to be efficiently solvable and others highly intractable in each case we outline solution concepts that take parametric uncertainty into account in the process of decision making

a novel actor critic identifier architecture for approximate optimal control of uncertain nonlinear systems an online adaptive reinforcement learning based solution is developed for the infinite horizon optimal control problem for continuous time uncertain nonlinear systems a novel actor critic identifier aci is proposed to approximate the hamilton jacobi bellman equation using three neural network nn structures actor and critic nns approximate the optimal control and the optimal value function respectively and a robust dynamic neural network identifier asymptotically approximates the uncertain system dynamics an advantage of using the aci architecture is that learning by the actor critic and identifier is continuous and simultaneous without requiring knowledge of system drift dynamics convergence of the algorithm is analyzed using lyapunov based adaptive control methods a persistence of excitation condition is required to guarantee exponential convergence to a bounded region in the neighborhood of the optimal control and uniformly ultimately bounded uub stability of the closed loop system simulation results demonstrate the performance of the actor critic identifier method for approximate optimal control

solving controlled markov set chains with discounting via multipolicy improvement we consider markov decision processes mdps where the state transition probability distributions are not uniquely known but are known to belong to some intervals so called controlled markov set chains with infinite horizon discounted reward criteria we present formal methods to improve multiple policies for solving such controlled markov set chains our multipolicy improvement methods follow the spirit of parallel rollout and policy switching for solving mdps in particular these methods are useful for online control of markov set chains and for designing policy iteration pi type algorithms we develop a pi type algorithm and prove that it converges to an optimal policy

e mdps learning in varying environments in this paper e mdp models are introduced and convergence theorems are proven using the generalized mdp framework of szepesvari and littman using this model family we show that q learning is capable of finding near optimal policies in varying environments the potential of this new family of mdp models is illustrated via a reinforcement learning algorithm called event learning which separates the optimization of decision making from the controller we show that event learning augmented by a particular controller which gives rise to an e mdp enables near optimal performance even if considerable and sudden changes may occur in the environment illustrations are provided on the two segment pendulum problem

biased lattice gases with correlated equilibrium states the approach to and structure of the equilibrium state is studied for a 7 bit lattice gas with biased forward and backward transition rates by means of mean field theory and computer simulations if the rate constants obey the factorizability and the detailed balance conditions the occupations of different velocity directions are uncorrelated anh theorem is valid and a nonuniversal equilibrium state exists that depends explicitly on the transition rates in case the above conditions are not satisfied theh theorem is no longer valid and mean field theory also predicts nontrivial velocity correlations in postcollision states the simulations are mainly concentrated on the time dependence of pre and postcollision velocity correlations on a single node and on slowly increasing fluctuations that might indicate metastable behavior

a fluctuation formula for the nongalilean factor in lattice gas automata a lattice gas automaton lacks galilei invariance and equilibria of systems moving with a finite speed mod u mod are not simply related by a galilei transformation to the equilibrium distribution in the rest frame in the hydrodynamic description of low speed equilibria in lattice gas automata a factor g p appears in the nonlinear convective term delta g p puu of the navier stokes equation that differs from unity due to lack of galilei invariance for this non galilean factor an expression in terms of fluctuating quantities is derived a grand ensemble where the total momentum is fluctuating around a zero average the formula is valid as long as there exists a unique equilibrium state consequently the results can also be used for a direct simulation of g p in lattice gas models where the explicit form of the equilibrium distribution is not known such as in models that violated semi detailed balance

on bayesian analysis of mixtures with an unknown number of components with discussion new methodology for fully bayesian mixture analysis is developed making use of reversible jump markov chain monte carlo methods that are capable of jumping between the parameter subspaces corresponding to different numbers of components in the mixture a sample from the full joint distribution of all unknown variables is thereby generated and this can be used as a basis for a thorough presentation of many aspects of the posterior distribution the methodology is applied here to the analysis of univariate normal mixtures using a hierarchical prior model that offers an approach to dealing with weak prior information while avoiding the mathematical pitfalls of using improper priors in the mixture context

low viscosity lattice gases a class of lattice gas models are studied which are variants of the fchc model the aim is to achieve the highest possible reynolds coefficient inverse dimensionless viscosity for efficient simulations of the three dimensional incompressible navier stokes equations the models include an arbitrary number of rest particles and violation of semi detailed balance within the framework of the boltzmann approximation exact expressions are obtained for the reynolds coefficients the minimization of the viscosity is done by solving a hitchcock type optimization problem for the fine tuning of the collision rules when the number of rest particles exceeds one there is a range of densities at which the viscosity takes negative values various optimal models with up to 26 bits per node have been implemented on a cray 2 and their true transport coefficients have been measured with good accuracy fairly large discrepancies with boltzmann values are observed when semi detailed balance is violated in particular no negative viscosity is obtained still the best model has a reynolds coefficient of 13 5 twice that of the best previously implemented model and thus is about 16 times more efficient computationally suggestions are made for further improvements it is proposed to use models with very high reynolds coefficients for sub grid scale modeling of turbulent flows

cellular automaton fluids 1 basic theory continuum equations are derived for the large scale behavior of a class of cellular automaton models for fluids the cellular automata are discrete analogues of molecular dynamics in which particles with discrete velocities populate the links of a fixed array of sites kinetic equations for microscopic particle distributions are constructed hydrodynamic equations are then derived using the chapman enskog expansion slightly modified navier stokes equations are obtained in two and three dimensions with certain lattices viscosities and other transport coefficients are calculated using the boltzmann transport equation approximation some corrections to the equations of motion for cellular automaton fluids beyond the navier stokes order are given

testing the number of components in a normal mixture we demonstrate that under a theorem proposed by vuong the likelihood ratio statistic based on the kullback leibler information criterion of the null hypothesis that a random sample is drawn from a k 0 component normal mixture distribution against the alternative hypothesis that the sample is drawn from a k 1 component normal mixture distribution is asymptotically distributed as a weighted sum of independent chi squared random variables with one degree of freedom under general regularity conditions we report simulation studies of two cases where we are testing a single normal versus a two component normal mixture and a two component normal mixture versus a three component normal mixture an empirical adjustment to the likelihood ratio statistic is proposed that appears to improve the rate of convergence to the limiting distribution

adaptive aggregation methods for infinite horizon dynamic programming a class of iterative aggregation algorithms for solving infinite horizon dynamic programming problems is proposed the idea is to interject aggregation iterations in the course of the usual successive approximation method an important feature that sets this method apart from earlier ones is that the aggregate groups of states change adaptively from one aggregation iteration to the next depending on the progress of the computation this allows acceleration of convergence in difficult problems involving multiple ergodic classes for which methods using fixed groups of aggregate states are ineffective no knowledge of special problem structure is utilized by the algorithms