a network algorithm for performing fisher s exact test in r c contingency tables abstract an exact test of significance of the hypothesis that the row and column effects are independent in an r c contingency table can be executed in principle by generalizing fisher s exact treatment of the 2 2 contingency table each table in a conditional reference set of r c tables with fixed marginal sums is assigned a generalized hypergeometric probability the significance level is then computed by summing the probabilities of all tables that are no larger on the probability scale than the observed table however the computational effort required to generate all r c contingency tables with fixed marginal sums severely limits the use of fisher s exact test a novel technique that considerably extends the bounds of computational feasibility of the exact test is proposed here the problem is transformed into one of identifying all paths through a directed acyclic network that equal or exceed a fixed length some interesting new optimization theorems are developed in the process the numer

a time aggregation approach to markov decision processes we propose a time aggregation approach for the solution of infinite horizon average cost markov decision processes via policy iteration in this approach policy update is only carried out when the process visits a subset of the state space as in state aggregation this approach leads to a reduced state space which may lead to a substantial reduction in computational and storage requirements especially for problems with certain structural properties however in contrast to state aggregation which generally results in an approximate model due to the loss of markov property time aggregation suffers no loss of accuracy because the markov property is preserved single sample path based estimation algorithms are developed that allow the time aggregation approach to be implemented on line for practical systems some numerical and simulation examples are presented to illustrate the ideas and potential computational savings

aggregation of the policy iteration method for nearly completely decomposable markov chains a steady state optimal control problem is considered for nearly completely decomposable markov chains in order to apply the policy iteration method of r a howard dynamic programming and markov processes cambridge ma mit press 1960 a high dimensional ill conditioned system of algebraic equations must be solved in the value determination step although algorithms exist for aggregation of the steady state probability distribution problem they only provide methods for computing the cost not the dual variables using a singular perturbation approach an aggregation method for the value determination equation is developed the aggregation method is developed in three steps first a class of similarity transformations that transform the system into a singularly perturbed form is developed second an aggregation method to compute the steady state probability distribution is derived third this aggregation method is applied to the value determination step of howard s method 1960

approximately exact inference for the common odds ratio in several 2 2 tables abstract the conditional maximum likelihood estimator mle of the common odds ratio in a sequence of independent 2 2 tables is known to be superior to the mantel haenszel estimator in terms of asymptotic efficiency and has the further advantage that its exact distribution is known however a long standing barrier to the widespread use of this estimator has been computational intractability in particular the calculation of significance levels confidence sets and power based on the exact distribution requires fast and efficient algorithms an important class of such algorithms form the basis of statxact a software package able to solve various aspects of the exact inference problem for a sequence of several 2 2 tables in real time we provide an alternative methodology by developing several useful saddlepoint approximations to the exact distribution of the conditional mle the approximations are derived from an interesting representation for hypergeometric random variables as a sum of independent

generalized monte carlo significance tests summary simple monte carlo significance testing has many applications particularly in the preliminary analysis of spatial data the method requires the value of the test statistic to be ranked among a random sample of values generated according to the null hypothesis however there are situations in which a sample of values can only be conveniently generated using a markov chain initiated by the observed data so that independence is violated this paper describes two methods that overcome the problem of dependence and allow exact tests to be carried out the methods are applied to the rasch model to the finite lattice ising model and to the testing of association between spatial processes power is discussed in a simple case

an optimal one way multigrid algorithm for discrete time stochastic control the numerical solution of discrete time stationary infinite horizon discounted stochastic control problems is considered for the case where the state space is continuous and the problem is to be solved approximately within a desired accuracy after a discussion of problem discretization the authors introduce a multigrid version of the successive approximation algorithm that proceeds one way from coarse to fine grids and analyze its computational requirements as a function of the desired accuracy and of the discount factor they also study the effects of a certain mixing ergodicity condition on the algorithm s performance it is shown that the one way multigrid algorithm improves upon the complexity of its single grid variant and is in a certain sense optimal

markov decision processes with fractional costs certain methods for constructing embedded markov decision processes mdps lead to performance measures that are the ratio of two long run averages for such mdps with finite state and action spaces and under an ergodicity assumption this note presents algorithms for computing optimal policies based on policy iterations linear programming value iterations and q learning

efficient monte carlo methods for conditional logistic regression abstract exact inference for the logistic regression model is based on generating the permutation distribution of the sufficient statistics for the regression parameters of interest conditional on the sufficient statistics for the remaining nuisance parameters despite the availability of fast numerical algorithms for the exact computations there are numerous instances where a data set is too large to be analyzed by the exact methods yet too sparse or unbalanced for the maximum likelihood approach to be reliable what is needed is a monte carlo alternative to the exact conditional approach which can bridge the gap between the exact and asymptotic methods of inference the problem is technically hard because conventional monte carlo methods lead to massive rejection of samples that do not satisfy the linear integer constraints of the conditional distribution we propose a network sampling approach to the monte carlo problem that eliminates rejection entirely its advantages over alternative saddlepoint

technical communique a unified approach to markov decision problems and performance sensitivity analysis we propose a simple approach that provides a unified formulation for the performance sensitivity analysis of markov chains and markov decision problems with both infinite horizon average cost and discounted performance criteria

practical use of higher order asymptotics for multiparameter exponential families recently developed asymptotics based on saddlepoint methods provide important practical methods for multiparameter exponential families especially in generalized linear models the aim here is to clarify and explore these attention is restricted to tests and confidence intervals regarding a single parametric function which can be represented as a natural parameter of a full rank exponential family excellent approximations to exact conditional inferences are often available in terms of simple adjustments to the signed square root of the likelihood ratio statistic

the control of a two level markov decision process by time aggregation the solution of markov decision processes mdps often relies on special properties of the processes for two level mdps the difference in the rates of state changes of the upper and lower levels has led to limiting or approximate solutions of such problems in this paper we solve a two level mdp without making any assumption on the rates of state changes of the two levels we first show that such a two level mdp is a non standard one where the optimal actions of different states can be related to each other then we give assumptions conditions under which such a specially constrained mdp can be solved by policy iteration we further show that the computational effort can be reduced by decomposing the mdp a two level mdp with m upper level states can be decomposed into one mdp for the upper level and m to m m 1 mdps for the lower level depending on the structure of the two level mdp the upper level mdp is solved by time aggregation a technique introduced in a recent paper cao x r ren z y bhatnagar s fu m marcus s 2002 a time aggregation approach to markov decision processes automatica 38 6 929 943 and the lower level mdps are solved by embedded markov chains

unbiased testing in exponential family regression let x ij z i i 1 2 k j 1 2 n i be independent observations such that z i is a fixed r x 1 vector r can be 0 no z s observed or 1 2 k 1 and x ij is distributed according to a one parameter exponential family which is log concave with natural parameter i we test the hypothesis that z where 1 k z is the matrix whose ith row is z i and 1 r is a vector of parameters we focus on r 2 and z i 1 z i i 1 2 k z i z i 1 the null hypothesis on hand is thus of the form i 1 2 z i in such a case the model under the null hypothesis becomes logistic regression in the binomial case poisson regression in the poisson case and linear regression in the normal case we consider mostly the one sided alternative that the second order differences of the natural parameters are nonnegative such testing problems test goodness of fit vs alternatives in which the natural parameters behave in a convex way we find classes of tests that are unbiased and that lie in a complete class we also note that every admissible test of constant size is unbiased in some discrete situations we find the minimal complete class of unbiased admissible tests generalizations and examples are given

a new value iteration method for the average cost dynamic programming problem we propose a new value iteration method for the classical average cost markovian decision problem under the assumption that all stationary policies are unichain and that furthermore there exists a state that is recurrent under all stationary policies this method is motivated by a relation between the average cost problem and an associated stochastic shortest path problem contrary to the standard relative value iteration our method involves a weighted sup norm contraction and for this reason it admits a gauss seidel implementation computational tests indicate that the gauss seidel version of the new method substantially outperforms the standard method for difficult problems

event based optimization of markov systems recent research indicates that markov decision processes mdps and perturbation analysis pa based optimization can be derived easily from two fundamental performance sensitivity formulas with this sensitivity point of view an event based optimization approach including event based sensitivity analysis and event based policy iteration was proposed via an example by x r cao discrete event dyn syst theory appl vol 15 pp 169 197 2005 this approach utilizes the special feature of a system and illustrates how the potentials can be aggregated using the special feature the approach applies to many practical problems that do not fit well the standard mdp formulation this note provides a mathematical formulation and proves the main results for this approach

an importance sampling algorithm for exact conditional tests in log linear models summary a simple but quite general simulation method for conducting exact conditional lackof fit tests in log linear models is proposed our monte carlo approximation utilises an importance sampling method motivated by the crude normal approximation to the poisson distribution examples considered include tests of quasi symmetry and related models for square tables and tests concerning higher order interactions in multi way tables the method is competitive with direct simulation from the exact conditional distribution when this is feasible and outperforms alternative monte carlo procedures when direct simulation is infeasible provided the number of degrees of freedom of the test is not too large extension of the method to tests against non saturated alternatives is straightforward and is briefly discussed and illustrated

computing distributions for exact logistic regression abstract logistic regression is a commonly used technique for the analysis of retrospective and prospective epidemiological and clinical studies with binary response variables usually this analysis is performed using large sample approximations when the sample size is small or the data structure sparse the accuracy of the asymptotic approximations is in question on other occasions singularity of the covariance matrix of parameter estimates precludes asymptotic analysis under these circumstances use of exact inferential procedures would seem to be a prudent alternative cox 1970 showed that exact inference on the parameters of a logistic model with binary response requires consideration of the distribution of sufficient statistics for these parameters to date however resorting to the exact method has not been computationally feasible except in a few special situations this article presents an efficient recursive algorithm that generates the joint and conditional distributions of the sufficient

sequential importance sampling for multiway tables we describe an algorithm for the sequential sampling of entries in multiway contingency tables with given constraints the algorithm can be used for computations in exact conditional inference to justify the algorithm a theory relates sampling values at each step to properties of the associated toric ideal using computational commutative algebra in particular the property of interval cell counts at each step is related to exponents on lead indeterminates of a lexicographic grobner basis also the approximation of integer programming by linear programming for sampling is related to initial terms of a toric ideal we apply the algorithm to examples of contingency tables which appear in the social and medical sciences the numerical results demonstrate that the theory is applicable and that the algorithm performs well

accelerating the convergence of value iteration by using partial transition functions abstract this work proposes an algorithm that makes use of partial information to improve the convergence properties of the value iteration algorithm in terms of the overall computational complexity the algorithm iterates on a series of increasingly refined approximate models that converges to the true model according to an optimal linear rate which coincides with the convergence rate of the original value iteration algorithm the paper investigates the properties of the proposed algorithm and features a series of switchover queue examples which illustrates the efficacy of the method

exact inference for categorical data recent advances and continuing controversies methods for exact small sample analyses with categorical data have been increasingly well developed in recent years a variety of exact methods exist primarily using the approach that eliminates unknown parameters by conditioning on their sufficient statistics in addition a variety of algorithms now exist for implementing the methods this paper briefly summarizes the exact approaches and describes recent developments controversy continues about the appropriateness of some exact methods primarily relating to their conservative nature because of discreteness this issue is examined for two simple problems in which discreteness can be severe interval estimation of a proportion and the odds ratio in general adjusted exact methods based on the mid p value seem a reasonable way of reducing the severity of this problem

permutation distributions via generating functions with applications to sensitivity analysis of discrete data abstract generating functions provide a simple and elegant way to describe probability or frequency distributions of discrete statistics and in particular permutation distributions they are also a computational tool many efficient algorithms including those described as fast fourier transform methods network methods and multiple shift methods are different implementations of the recursions needed to evaluate generating functions efficiently our goals here are twofold first we make the relationship between these efficient methods and generating functions explicit this establishes a language for looking at other questions in randomization exact inference and may help in finding more efficient implementations second we propose methods to examine the sensitivity of results of exact analysis of discrete data to small perturbations in the data specifically we consider two settings how the analysis would change if one outcome changed and how the analysis would change if one observation was adde

stability and optimality of a multi product production and storage system under demand uncertainty this work develops a discrete event model for a multi product multi stage production and storage p s problem subject to random demand the intervention problem consists of three types of possible decisions made at the end of one stage which depend on the observed demand or lack of for each item i to proceed further with the production of the same product ii to proceed with the production of another product or iii to halt the production the intervention problem is formulated in terms of dynamic programming dp operators and each possible solution induces an homogeneous markov chain that characterizes the dynamics however solving directly the dp problem is not a viable task in situations involving a moderately large number of products with many production stages and the idea of the paper is to detach from strict optimality with monitored precision and rely on stability the notion of stochastic stability brought to bear requires a finite set of positive recurrent states and the paper derives necessary and sufficient conditions for a policy to induce such a set in the studied p s problem an approximate value iteration algorithm is proposed which applies to the broader class of control problems described by homogeneous markov chains that satisfy a structural condition pointed out in the paper this procedure iterates in a finite subset of the state space circumventing the computational burden of standard dynamic programming to benchmark the approach the proposed algorithm is applied to a simple two product p s system

simulation based optimization of markov reward processes this paper proposes a simulation based algorithm for optimizing the average reward in a finite state markov reward process that depends on a set of parameters as a special case the method applies to markov decision processes where optimization takes place within a parametrized set of policies the algorithm relies on the regenerative structure of finite state markov processes involves the simulation of a single sample path and can be implemented online a convergence result with probability 1 is provided

algebraic algorithms for sampling from conditional distributions we construct markov chain algorithms for sampling from discrete exponential families conditional on a sufficient statistic examples include contingency tables logistic regression and spectral analysis of permutation data the algorithms involve computations in polynomial rings using grobner bases

approximate dynamic programming via direct search in the space of value function approximations this paper deals with approximate value iteration avi algorithms applied to discounted dynamic programming dp problems for a fixed control policy the span semi norm of the so called bellman residual is shown to be convex in the banach space of candidate solutions to the dp problem this fact motivates the introduction of an avi algorithm with local search that seeks to minimize the span semi norm of the bellman residual in a convex value function approximation space the novelty here is that the optimality of a point in the approximation architecture is characterized by means of convex optimization concepts and necessary and sufficient conditions to local optimality are derived the procedure employs the classical avi algorithm direction bellman residual combined with a set of independent search directions to improve the convergence rate it has guaranteed convergence and satisfies at least the necessary optimality conditions over a prescribed set of directions to illustrate the method examples are presented that deal with a class of problems from the literature and a large state space queueing problem setting

adaptive control of markov chains with average cost we present an adaptive control scheme for the control of markov chains to minimize long run average cost when the system transition and reward structures are unknown q factors are estimated along a single sample path of the system and control actions are applied based on the latest estimates we prove that an optimal policy is obtained asymptotically with probability one more importantly we prove that optimal system performance is achieved as well which means that the performance of the system can not be bettered even if the system transition and reward structures are known an example is given to illustrate our adaptive control scheme

computing exact distributions for polytomous response data abstract this article presents an efficient method for computing exact conditional distributions of the sufficient statistics for the parameters of four polytomous response models for nominal response two baseline category logit models and for ordinal response two adjacent categories logit models are considered the method consists of recursive generation of the joint distribution of the sufficient statistics augmented by a technique to slice off portions of the evolving sample space so as to eventually yield the required conditional distribution two actual data sets are analyzed to illustrate the method

aggregation bounds in stochastic linear programming stochastic linear programs become extremely large and complex as additional uncertainties and possible future outcomes are included in their formulation row and column aggregation can significantly reduce this complexity but the solutions of the aggregated problem only provide an approximation of the true solution in this paper error bounds on the value of the optimal solution of the original problem are obtained from the solution of the aggregated problem these bounds apply for aggregation of both random variables and time periods

a diagnostic test for the mixing distribution in a generalised linear mixed model summary we introduce a diagnostic test for the mixing distribution in a generalised linear mixed model the test is based on the difference between the marginal maximum likelihood and conditional maximum likelihood estimators of a subset of the fixed effects in the model we derive the asymptotic variance of this difference and propose a test statistic that has a limiting chi squared distribution under the null hypothesis that the mixing distribution is correctly specified this strategy uses an idea presented by hausman 1978 who considered analogous tests for the linear mixed model an important advantage of the methods outlined here is that the resulting diagnostic test is easily implemented in commercial software we illustrate the method by applying it to data from a clinical trial investigating the effect of hormonal contraceptives in women

potential based online policy iteration algorithms for markov decision processes performance potentials play a crucial role in performance sensitivity analysis and policy iteration of markov decision processes the potentials can be estimated on a single sample path of a markov process in this paper we propose two potential based online policy iteration algorithms for performance optimization of markov systems the algorithms are based on online estimation of potentials and stochastic approximation we prove that with these two algorithms the optimal policy can be attained after a finite number of iterations a simulation example is given to illustrate the main ideas and the convergence rates of the algorithms

intrinsic construction of invariant functions on simple lie algebras abstract an algorithm for constructing primitive adjoint invariant functions on a complex simple lie algebra is presented the construction is intrinsic in the sense that it does not resort to any representation a primitive invariant function on the whole lie algebra is obtained by lifting a coordinate function on a kostant slice of the lie algebra such an intrinsic construction of invariant functions is most useful for the bigger exceptional lie algebras such as the e s the maple implementation of this algorithm is outlined at the end and will be applied to these exceptional lie algebras in a future work

testing logistic regression coefficients with clustered data and few positive outcomes applications frequently involve logistic regression analysis with clustered data where there are few positive outcomes in some of the independent variable categories for example an application is given here that analyzes the association of asthma with various demographic variables and risk factors using data from the third national health and nutrition examination survey a weighted multi stage cluster sample although there are 742 asthma cases in all out of 18 395 individuals for one of the categories of one of the independent variables there are only 25 asthma cases out of 695 individuals generalized wald and score hypothesis tests which use appropriate cluster level variance estimators and a bootstrap hypothesis test have been proposed for testing logistic regression coefficients with cluster samples when there are few positive outcomes simulations presented in this paper show that these tests can sometimes have either inflated or very conservative levels a simulation based method is proposed for testing logistic regression coefficients with cluster samples when there are few positive outcomes this testing methodology is shown to compare favorably with the generalized wald and score tests and the bootstrap hypothesis test in terms of maintaining nominal levels the proposed method is also useful when testing goodness of fit of logistic regression models using deciles of risk tables published in 2007 by john wiley sons ltd

solving average cost markov decision processes by means of a two phase time aggregation algorithm abstract this paper introduces a two phase approach to solve average cost markov decision processes which is based on state space embedding or time aggregation in the first phase time aggregation is applied for policy optimization in a prescribed subset of the state space and a novel result is applied to expand the evaluation to the whole state space this evaluation is then used in the second phase in a policy improvement step and the two phases are then alternated until convergence is attained some numerical experiments illustrate the results

an algorithm for exact logistic regression abstract an efficient algorithm is given for the exact logistic analysis of a single parameter hypothesis tests and confidence intervals for a logistic regression parameter are obtained the method provides for the control of the effect of a stratification factor when making inferences concerning the regression parameter

an iterative aggregation procedure for markov decision processes an iterative aggregation procedure is described for solving large scale finite state finite action markov decision processes mdps at each iteration an aggregate master problem and a sequence of smaller subproblems are solved the weights used to form the aggregate master problem are based on the estimates from the previous iteration each subproblem is a finite state finite action mdp with a reduced state space and unequal row sums global convergence of the algorithm is proven under very weak assumptions the proof relates this technique to other iterative methods that have been suggested for general linear programs

practical small sample asymptotics for regression problems abstract saddlepoint approximations are derived for sums of independent but not necessarily identically distributed random variables along with generalizations to estimating equations and multivariate problems these results are particularly useful for accurately approximating the distribution of regression coefficients general formulas are given for the distribution of the coefficients arising out of a generalized linear model with both canonical and noncanonical link functions we illustrate the case of logistic regression with a real data example and show how the gibbs sampler may be used to obtain confidence sets for each regression parameter based on the saddlepoint approximation

approximate receding horizon approach for markov decision processes average reward case abstract we consider an approximation scheme for solving markov decision processes mdps with countable state space finite action space and bounded rewards that uses an approximate solution of a fixed finite horizon sub mdp of a given infinite horizon mdp to create a stationary policy which we call approximate receding horizon control we first analyze the performance of the approximate receding horizon control for infinite horizon average reward under an ergodicity assumption which also generalizes the result obtained by white j oper res soc 33 1982 253 259 we then study two examples of the approximate receding horizon control via lower bounds to the exact solution to the sub mdp the first control policy is based on a finite horizon approximation of howard s policy improvement of a single policy and the second policy is based on a generalization of the single policy improvement for multiple policies along the study we also provide a simple alternative proof on the policy improvement for countable state space we finally discuss practical implementations of these schemes via simulation

sampling for conditional inference on case control data summary the problem of exact conditional inference for discrete multivariate case control data has two forms the first is grouped case control data where monte carlo computations can be done using the importance sampling method of booth and butler 1999 biometrika86 321 332 or a proposed alternative sequential importance sampling method the second form is matched case control data for this analysis we propose a new exact sampling method based on the conditional poisson distribution for conditional testing with one binary and one integral ordered covariate this method makes computations on data sets with large numbers of matched sets fast and accurate we provide detailed derivation of the constraints and conditional distributions for conditional inference on grouped and matched data the methods are illustrated on several new and old data sets

approximate conditional inference in generalized linear models on developpe des approximations precises et facilement calculables pour les densites conditionnelles et les distributions des statistiques exhaustives dans les modeles lineaires generalises avec des fonctions de liaison canoniques

liouville integrability of a class of integrable spin calogero moser systems and exponents of simple lie algebras in previous work we introduced a class of integrable spin calogero moser systems associated with the classical dynamical r matrices with spectral parameter as classified by etingof and varchenko for simple lie algebras here the main purpose is to establish the liouville integrability of these systems by a uniform method based on evaluating the primitive invariants of chevalley on the lax operators with spectral parameter as part of our analysis we will develop several results concerning the algebra of invariant polynomials on simple lie algebras and their expansions

optimal kullback leibler aggregation via information bottleneck in this paper we present a method for reducing a regular discrete time markov chain dtmc to another dtmc with a given typically much smaller number of states the cost of reduction is defined as the kullback leibler divergence rate between a projection of the original process through a partition function and a dtmc on the correspondingly partitioned state space finding the reduced model with minimal cost is computationally expensive as it requires an exhaustive search among all state space partitions and an exact evaluation of the reduction cost for each candidate partition our approach deals with the latter problem by minimizing an upper bound on the reduction cost instead of minimizing the exact cost the proposed upper bound is easy to compute and it is tight if the original chain is lumpable with respect to the partition then we express the problem in the form of information bottleneck optimization and propose using the agglomerative information bottleneck algorithm for searching a suboptimal partition greedily rather than exhaustively the theory is illustrated with examples and one application scenario in the context of modeling bio molecular interactions

lao a heuristic search algorithm that finds solutions with loops classic heuristic search algorithms can find solutions that take the form of a simple path a a tree or an acyclic graph ao in this paper we describe a novel generalization of heuristic search called lao that can find solutions with loops we show that lao can be used to solve markov decision problems and that it shares the advantage heuristic search has over dynamic programming for other classes of problems given a start state it can find an optimal solution without evaluating the entire state space 2001 elsevier science b v all rights reserved

approximate conditional inference in exponential families via the gibbs sampler abstract this article presents the gibbs skovgaard algorithm for approximate frequentist inference the method makes use of the double saddlepoint approximation of skovgaard to the conditional cumulative distribution function of a sufficient statistic given the remaining sufficient statistics this approximation is then used in the gibbs sampler to generate a markov chain the equilibrium distribution of this chain approximates the joint distribution of the sufficient statistics associated with the parameters of interest conditional on the observed values of the sufficient statistics associated with the nuisance parameters this gibbs skovgaard algorithm is applied to the cases of logistic and poisson regression

an analysis of stochastic shortest path problems we consider a stochastic version of the classical shortest path problem whereby for each node of a graph we must choose a probability distribution over the set of successor nodes so as to reach a certain destination node with minimum expected cost the costs of transition between successive nodes can be positive as well as negative we prove natural generalizations of the standard results for the deterministic shortest path problem and we extend the corresponding theory for undiscounted finite state markovian decision problems by removing the usual restriction that costs are either all nonnegative or all nonpositive

brief paper average cost temporal difference learning we propose a variant of temporal difference learning that approximates average and differential costs of an irreducible aperiodic markov chain approximations are comprised of linear combinations of fixed basis functions whose weights are incrementally updated during a single endless trajectory of the markov chain we present a proof of convergence with probability 1 and a characterization of the limit of convergence we also provide a bound on the resulting approximation error that exhibits an interesting dependence on the mixing time of the markov chain the results parallel previous work by the authors involving approximations of discounted cost to go

performance loss bounds for approximate value iteration with state aggregation we consider approximate value iteration with a parameterized approximator in which the state space is partitioned and the optimal cost to go function over each partition is approximated by a constant we establish performance loss bounds for policies derived from approximations associated with fixed points these bounds identify benefits to using invariant distributions of appropriate policies as projection weights such projection weighting relates to what is done by temporal difference learning our analysis also leads to the first performance loss bound for approximate value iteration with an average cost objective

performing the exact test of hardy weinberg proportion for multiple alleles summary the hardy weinberg law plays an important role in the field of population genetics and often serves as a basis for genetic inference because of its importance much attention has been devoted to tests of hardy weinberg proportions hwp over the decades it has long been recognized that largesample goodness of fit tests can sometimes lead to spurious results when the sample size and or some genotypic frequencies are small although a complete enumeration algorithm for the exact test has been proposed it is not of practical use for loci with more than a few alleles due to the amount of computation required we propose two algorithms to estimate the significance level for a test of hwp the algorithms are easily applicable to loci with multiple alleles both are remarkably simple and computationally fast relative efficiency and merits of the two algorithms are compared guidelines regarding their usage are given numerical examples are given to illustrate the practicality of the algorithms

an analysis of temporal difference learning with function approximation we discuss the temporal difference learning algorithm as applied to approximating the cost to go function of an infinite horizon discounted markov chain the algorithm we analyze updates parameters of a linear function approximator online during a single endless trajectory of an irreducible aperiodic markov chain with a finite or infinite state space we present a proof of convergence with probability one a characterization of the limit of convergence and a bound on the resulting approximation error furthermore our analysis is based on a new line of reasoning that provides new intuition about the dynamics of temporal difference learning in addition to proving new and stronger positive results than those previously available we identify the significance of online updating and potential hazards associated with the use of nonlinear function approximators first we prove that divergence may occur when updates are not based on trajectories of the markov chain this fact reconciles positive and negative results that have been discussed in the literature regarding the soundness of temporal difference learning second we present an example illustrating the possibility of divergence when temporal difference learning is used in the presence of a nonlinear function approximator

data analytic methods for matched case control studies the recent introduction of complex multivariate statistical models in matched case control studies is a mixed blessing their use can lead to a better understanding of the way in which many variables contribute to the risk of disease on the other hand these powerful methods can obscure salient features in the data that might have been detected by other less sophisticated methods this shortcoming is due to a lack of support methodology for the routine use of these models satisfactory computation of estimated relative risks and their standard errors is not sufficient justification for the fitted model goodness of fit must be examined if inferences are to be trusted this paper is concerned with the analysis of matched case control studies with logistic models analogies of these models to linear regression models are emphasized in particular basic concepts such as analysis of variance multiple correlation coefficient one degree of freedom tests and residual analysis are discussed the fairly new field of regression diagnostics is also introduced all procedures are illustrated on a study of bladder cancer in males

equivalence between conditional and mixture approaches to the rasch model and matched case control studies with applications analyses of data using rasch models including the special case of matched case control studies are common applications of conditional likelihood in which the usual inferential procedures are applied only after conditioning on an approximately ancillary statistic another common approach to the analysis of rasch models is to integrate the nuisance parameters over a mixing distribution using the marginal likelihood obtained as the basis for inference we show that the full conditional likelihood can always be obtained exactly via the marginal approach given a particular choice of mixing distribution and derive necessary and sufficient conditions for the two approaches to agree previous work has shown that with sufficient flexibility in the mixing distribution the maxima of the marginal and conditional likelihoods will be equivalent under concordance criteria our argument requires no such criteria and for any dataset guarantees the equivalence of the whole of the two likelihoods not just their maxi

small sample confidence regions in exponential families this article presents an algorithm for small sample conditional confidence regions for two or more parameters for any discrete regression model in the generalized linear interactive model family regions are constructed by careful inversion of conditional hypothesis tests this method presupposes the use of approximate or exact techniques for enumerating the sample space for some components of the vector of sufficient statistics conditional on other components such enumeration may be performed exactly or by exact or approximate monte carlo including the algorithms of kolassa and tanner 1994 journal of the american statistical association 89 697 702 1999 biometrics 55 246 251 this method also assumes that one can compute certain conditional probabilities for a fixed value of the parameter vector because of a property of exponential families one can use this set of conditional probabilities to directly compute the conditional probabilities associated with any other value of the vector of the parameters of interest this observation dramatically reduces the computational effort required to invert the hypothesis test to obtain the confidence region to construct a region with confidence level 1 alpha the algorithm begins with a grid of values for the parameters of interest for each parameter vector on the grid corresponding to the current null hypothesis one transforms the initial set of conditional probabilities using exponential tilting and then calculates the p value for this current null hypothesis the confidence region is the set of parameter values for which the p value is at least alpha

lebesgue sampling based optimal control problems with time aggregation we formulate the lebesgue sampling based optimal control problem we show that the problem can be solved by the time aggregation approach in markov decision processes mdp theory policy iteration based and reinforcement learning based methods are developed for the optimal policies both analytical solutions and sample path based algorithms are given compared to the periodic sampling scheme the lebesgue sampling scheme improves system performance

approximate dynamic programming with a fuzzy parameterization dynamic programming dp is a powerful paradigm for general nonlinear optimal control computing exact dp solutions is in general only possible when the process states and the control actions take values in a small discrete set in practice it is necessary to approximate the solutions therefore we propose an algorithm for approximate dp that relies on a fuzzy partition of the state space and on a discretization of the action space this fuzzy q iteration algorithm works for deterministic processes under the discounted return criterion we prove that fuzzy q iteration asymptotically converges to a solution that lies within a bound of the optimal solution a bound on the suboptimality of the solution obtained in a finite number of iterations is also derived under continuity assumptions on the dynamics and on the reward function we show that fuzzy q iteration is consistent i e that it asymptotically obtains the optimal solution as the approximation accuracy increases these properties hold both when the parameters of the approximator are updated in a synchronous fashion and when they are updated asynchronously the asynchronous algorithm is proven to converge at least as fast as the synchronous one the performance of fuzzy q iteration is illustrated in a two link manipulator control problem

an exact trend test for correlated binary data the problem of testing a dose response relationship in the presence of exchangeably correlated binary data has been addressed using a variety of models most commonly used approaches are derived from likelihood or generalized estimating equations and rely on large sample theory to justify their inferences however while earlier work has determined that these methods may perform poorly for small or sparse samples there are few alternatives available to those faced with such data we propose an exact trend test for exchangeably correlated binary data when groups of correlated observations are ordered this exact approach is based on an exponential model derived by molenberghs and ryan 1999 and ryan and molenberghs 1999 and provides natural analogues to fisher s exact test and the binomial trend test when the data are correlated we use a graphical method with which one can efficiently compute the exact tail distribution and apply the test to two examples

the conditional distribution of goodness of fit statistics for discrete data abstract i consider the distribution of pearson s statistic and of the likelihood ratio goodness of fit statistic for discrete data in the important case where the data are extensive but sparse it is argued that the appropriate reference distribution is conditional on the sufficient statistic for the unknown regression parameters the first three conditional asymptotic cumulants are derived by edgeworth expansion and these are used for the computation of tail probabilities the principal advantage of the limit considered here as opposed to the more usual x 2 limit is that the cell counts need not be large

aggregation and disaggregation techniques and methodology in optimization a fundamental issue in the use of optimization models is the tradeoff between the level of detail and the ease of using and solving the model aggregation and disaggregation techniques have proven to be valuable tools for manipulating data and determining the appropriate policies to employ for this tradeoff furthermore aggregation and disaggregation techniques offer promise for solving large scale optimization models supply a set of promising methodologies for studying the underlying structure of both univariate and multivariate data sets and provide a set of tools for manipulating data for different levels of decision makers in this paper we develop a general framework for aggregation and disaggregation methodology survey previous work regarding aggregation and disaggregation techniques for optimization problems illuminate the appropriate role of aggregation and disaggregation methodology for optimization applications and propose future research directions

performance gradient estimation for the very large finite markov chains using an embedded markov chain the steady state performance gradient estimation for very large markov chains is decomposed into two smaller problems one solved by the stochastic recursive sr method and the other by the likelihood ratio lr method the combination named sr lr can have several magnitudes of lower space requirement than the sr and several magnitudes of lower time consumptions than the lr both the lr and the sr are the extreme cases of the combined algorithm the authors demonstrate the sr lr methods on markov chains with hundreds of millions of states which are created using queuing networks

computational tools for exact conditional logistic regression logistic regression analyses are often challenged by the inability of unconditional likelihood based approximations to yield consistent valid estimates and p values for model parameters this can be due to sparseness or separability in the data conditional logistic regression though useful in such situations can also be computationally unfeasible when the sample size or number of explanatory covariates is large we review recent developments that allow efficient approximate conditional inference including monte carlo sampling and saddlepoint approximations we demonstrate through real examples that these methods enable the analysis of significantly larger and more complex data sets we find in this investigation that for these moderately large data sets monte carlo seems a better alternative as it provides unbiased estimates of the exact results and can be executed in less cpu time than can the single saddlepoint approximation moreover the double saddlepoint approximation while computationally the easiest to obtain offers little practical advantage it produces unreliable results and cannot be computed when a maximum likelihood solution does not exist

incremental value iteration for time aggregated markov decision processes a value iteration algorithm for time aggregated markov decision processes mdps is developed to solve problems with large state spaces the algorithm is based on a novel approach which solves a time aggregated mdp by incrementally solving a set of standard mdps therefore the algorithm converges under the same assumption as standard value iteration such assumption is much weaker than that required by the existing time aggregated value iteration algorithm the algorithms developed in this paper are also applicable to mdps with fractional costs

noncommutative and commutative integrability of generic toda flows in simple lie algebras 1 1 the toda lattice equation introduced by toda as a hamilton equation describing the motion of the system of particles on the line with an exponential interaction between closest neighbours gave rise to numerous important generalizations and helped to discover many of the exciting phenomena in the theory of integrable equations in flaschka s variables f the finite non periodic toda lattice describes an isospectral evolution on the set of tri diagonal matrices in sl n it was explicitly solved and shown to be completely integrable by moser mo in his paper k1 kostant comprehensively studied the generalization of toda lattice that evolves on the set of tri diagonal elements of a semisimple lie algebra g which also turned out to be completely integrable with poisson commuting integrals being provided by the chevalley invariants of the algebra moreover in this paper as well as in the works by ol shanetsky and perelomov op reyman and semenov tian shansky rsts1 symes sy the method of the explicit integration of the toda equations was extended to the case when evolution takes place on the dual space of the borel subalgebra of g this space is foliated into symplectic leaves of different dimensions and the natural question is what can be said about the liouville complete integrability of the toda flows on each of these leaves in the particular case of generic symplectic leaves in sl n the complete integrability was proved by deift li nanda and tomei dlnt this paper was motivated by the work dlnt and its lie algebraic interpretation proposed in s1 s2 efs our main result is the following

the octic e8 invariant we give an explicit expression for the primitive e8 invariant tensor with eight symmetric indices the result is presented in a manifestly spin 16 z2 covariant notation

an empirical assessment of saddlepoint approximations for testing a logistic regression parameter saddlepoint methods provide quick and easy approximations to significance levels for conditional tests of logistic regression parameters we evaluate the accuracies of saddlepoint approximations for three well known conditional tests bartlett s test for no three factor interaction in a 2 x 2 x 2 table the test for trend in a series of probabilities and the exact test of no association in stratified 2 x 2 tables with a common odds ratio general recommendations are suggested regarding the use of saddlepoint approximations for exact conditional significance levels

assessing robustness of intrinsic tests of independence in two way contingency tables for testing nested hypotheses from a bayesian standpoint a desirable condition is that the prior for the alternative model concentrates mass around the smaller or null model for testing independence in contingency tables the intrinsic priors satisfy this requirement furthermore the degree of concentration of the priors is controlled by a discrete parameter t the training sample size which plays an important role in the resulting answer in this article we report on the robustness of the tests of independence for small or moderate sample sizes in contingency tables with respect to intrinsic priors with different degrees of concentration around the null we compare these tests to frequentist tests and other robust bayes tests for large sample sizes robustness is achieved because the intrinsic bayesian tests are consistent examples using real and simulated data are given supplemental materials technical details and data sets are available online

monte carlo exact conditional tests for log linear and logistic models summary the form of the exact conditional distribution of a sufficient statistic for the interest parameters given a sufficient statistic for the nuisance parameters is derived for a generalized linear model with canonical link general results for log linear and logistic models are given a gibbs sampling approach for generating from the conditional distribution is proposed which enables monte carlo exact conditional tests to be performed examples include tests for goodness of fit of the all two way interaction model for a 28 table and of a simple logistic model tests against non saturated alternatives are also considered

the linear programming approach to approximate dynamic programming the curse of dimensionality gives rise to prohibitive computational requirements that render infeasible the exact solution of large scale stochastic control problems we study an efficient method based on linear programming for approximating solutions to such problems the approach fits a linear combination of pre selected basis functions to the dynamic programming cost to go function we develop error bounds that offer performance guarantees and also guide the selection of both basis functions and state relevance weights that influence quality of the approximation experimental results in the domain of queueing network control provide empirical support for the methodology

on state aggregation to approximate complex value functions in large scale markov decision processes many small electronic devices such as cell phones and wireless sensors have restrictive memory space computing power and battery the pervasive applications of these devices in industry military and our daily lives require simple policies that are easy to implement and can be executed in a reasonably short time the markov decision process mdp provides a general framework for these policy optimization problems with complexity constraint in many cases we can use a powerful computer to find the optimal or a good policy and the value function first and then approximate by a simple one the approximation usually depends on heuristics or experiences because the relationship between the complexity of a function and the approximation error is not clear in general in this paper we assume the optimal value function is known or a reasonably good estimate is available and consider how to approximate a complex value function due to the broad application of state aggregation in the large scale mdp we focus on piecewise constant approximate value functions and use the number of aggregated states to measure the complexity of a value function we quantify how the complexity of a value function affects the approximation error when the optimal value function is known for sure we develop an algorithm that finds the best simple state aggregation within polynomial time when we have estimates of the optimal value function we apply ordinal optimization to find good simple state aggregations with high probability the algorithms are demonstrated on a node activation policy optimization problem in wireless sensor network we hope this work can shed some insight on how to find simple policies with good performances

a comparative investigation of methods for logistic regression with separated or nearly separated data in logistic regression analysis of small or sparse data sets results obtained by classical maximum likelihood methods cannot be generally trusted in such analyses it may even happen that the likelihood meets the convergence criteria while at least one parameter estimate diverges to infinity this situation has been termed separation and it typically occurs whenever no events are observed in one of the two groups defined by a dichotomous covariate more generally separation is caused by a linear combination of continuous or dichotomous covariates that perfectly separates events from non events separation implies infinite or zero maximum likelihood estimates of odds ratios which are usually considered unrealistic i provide some examples of separation and near separation in clinical data sets and discuss some options to analyse such data including exact logistic regression analysis and a penalized likelihood approach both methods supply finite point estimates in case of separation profile penalized likelihood confidence intervals for parameters show excellent behaviour in terms of coverage probability and provide higher power than exact confidence intervals general advantages of the penalized likelihood approach are discussed

assessing the fit of the logistic regression model to individual matched sets of case control data we develop exact conditional methods for checking the fit of a logistic regression model to individual matched sets in case control studies justifications are given for preferring this approach to conventional maximum likelihood based methods data from two studies are used to illustrate the techniques

using randomization to break the curse of dimensionality this paper introduces random versions of successive approximations and multigrid algorithms for computing approximate solutions to a class of finite and infinite horizon markovian decision problems mdps we prove that these algorithms succeed in breaking the curse of dimensionality for a subclass of mdps known as discrete decision processes ddps

exact finite approximations of average cost countable markov decision processes for a countable state markov decision process we introduce an embedding which produces a finite state markov decision process the finite state embedded process has the same optimal cost and moreover it has the same dynamics as the original process when restricting to the approximating set the embedded process can be used as an approximation which being finite is more convenient for computation and implementation

semi markov decision problems and performance sensitivity analysis recent research indicates that markov decision processes mdps can be viewed from a sensitivity point of view and the perturbation analysis pa mdps and reinforcement learning rl are three closely related areas in optimization of discrete event dynamic systems that can be modeled as markov processes the goal of this paper is two fold first we develop the pa theory for semi markov processes smps and then we extend the aforementioned results about the relation among pa mdp and rl to smps in particular we show that performance sensitivity formulas and policy iteration algorithms of semi markov decision processes can be derived based on the performance potential and realization matrix both the long run average and discounted cost problems are considered this approach provides a unified framework for both problems and the long run average problem corresponds to the discounted factor being zero the results indicate that performance sensitivities and optimization depend only on first order statistics single sample path based implementations are discussed

adaptive aggregation methods for infinite horizon dynamic programming a class of iterative aggregation algorithms for solving infinite horizon dynamic programming problems is proposed the idea is to interject aggregation iterations in the course of the usual successive approximation method an important feature that sets this method apart from earlier ones is that the aggregate groups of states change adaptively from one aggregation iteration to the next depending on the progress of the computation this allows acceleration of convergence in difficult problems involving multiple ergodic classes for which methods using fixed groups of aggregate states are ineffective no knowledge of special problem structure is utilized by the algorithms