computationally feasible bounds for partially observed markov decision processes a partially observed markov decision process pomdp is a sequential decision problem where information concerning parameters of interest is incomplete and possible actions include sampling surveying or otherwise collecting additional information such problems can theoretically be solved as dynamic programs but the relevant state space is infinite which inhibits algorithmic solution this paper explains how to approximate the state space by a finite grid of points and use that grid to construct upper and lower value function bounds generate approximate nonstationary and stationary policies and bound the value loss relative to optimal for using these policies in the decision problem a numerical example illustrates the methodology

proportional likelihood ratio models for mean regression the proportional likelihood ratio model introduced in luo tsai 2012 is adapted to explicitly model the means of observations this is useful for the estimation of and inference on treatment effects particularly in designed experiments and allows the data analyst greater control over model specification and parameter interpretation

lower bounds for the heights of solutions of linear equations let k be a number eld of degree d we de ne as usual the eld height hk of a number k by hk v m k max 1 dv v here the v adic valuations v are normalized such that on q they coincide with either the standard absolute value or with the p adic absolute value p having p p p 1 moreover dv denotes the local degree the absolute height of then is de ned by h hk lehmer s conjecture says the following let be a nonzero algebraic number of degree d then if is not a root of unity we have

moment type estimators for the proportional likelihood ratio model with longitudinal data luo tsai biometrika 99 211 22 2012 proposed a proportional likelihood ratio model and discussed a maximum likelihood method for its parameter estimation in this paper we use this model as the marginal distribution to analyse longitudinal data where the maximum likelihood method is not directly applicable because the joint distribution is not fully specified we propose a moment type method that is an extension of the generalized estimating equation method the resulting estimators are consistent asymptotically normal and perform well in our simulation study

action elimination procedures for modified policy iteration algorithms this paper proposes bounds and action elimination procedures for policy iteration and modified policy iteration procedures to eliminate nonoptimal actions for one iteration and for all subsequent iterations are presented the implementation of these procedures is discussed and encouraging computational results are presented

the multiplicity of binary recurrences of the recurrence relation 0 1 it follows from the skolem mahler lech theorem that if u then one at least of and is a root of unity consequently we call um nondegenerate if none of these quantities is a root of unity in the thirties m ward conjectured that a nondegenerate binary recurrence sequence of integers um has multiplicity u 5 5 this conjecture was proved by kubota 4 kubota also proved that any nondegenerate binary recurrence um with elements um in a number eld k of degree d has multiplicity u 5 c d cf 5 beukers and tijdeman 2 established for this case the bound

partially observable markov decision processes a geometric technique and analysis this paper presents a novel framework for studying partially observable markov decision processes pomdps with finite state action observation sets and discounted rewards the new framework is solely based on future reward vectors associated with future policies which is more parsimonious than the traditional framework based on belief vectors it reveals the connection between the pomdp problem and two computational geometry problems i e finding the vertices of a convex hull and finding the minkowski sum of convex polytopes which can help solve the pomdp problem more efficiently the new framework can clarify some existing algorithms over both finite and infinite horizons and shed new light on them it also facilitates the comparison of pomdps with respect to their degree of observability as a useful structural result

on the diophantine equation gn x gm p x higher order recurrences let k be a field of characteristic 0 and let g n x n 0 be a linear recurring sequence of degree d in k x defined by the initial terms g 0 c d 1 k x and by the difference equation g n d x a d 1 x g n d 1 x a 0 x g n x for n 0 with a 0 a d 1 k x finally let p x be an element of k x in this paper we are giving fairly general conditions depending only on g 0 g d 1 on p and on a 0 a d 1 under which the diophantine equation g n x g m p x has only finitely many solutions n m z 2 n m 0 moreover we are giving an upper bound for the number of solutions which depends only on d this paper is a continuation of the work of the authors on this equation in the case of second order linear recurring sequences

state of the art a survey of partially observable markov decision processes theory models and algorithms this paper surveys models and algorithms dealing with partially observable markov decision processes a partially observable markov decision process pomdp is a generalization of a markov decision process which permits uncertainty regarding the state of a markov process and allows for state information acquisition a general framework for finite state and action pomdp s is presented next there is a brief discussion of the development of pomdp s and their relationship with other decision processes a wide range of models in such areas as quality control machine maintenance internal auditing learning and optimal stopping are discussed within the pomdp framework lastly algorithms for computing optimal solutions to pomdp s are presented

a proportional likelihood ratio model we propose a semiparametric proportional likelihood ratio model which is particularly suitable for modelling a nonlinear monotonic relationship between the outcome variable and a covariate this model extends the generalized linear model by leaving the distribution unspecified and has a strong connection with semiparametric models such as the selection bias model gilbert et al 1999 the density ratio model qin 1998 fokianos kaimi 2006 the single index model ichimura 1993 and the exponential tilt regression model rathouz gao 2009 a maximum likelihood estimator is obtained for the new model and its asymptotic properties are derived an example and simulation study illustrate the use of the model copyright 2012 oxford university press

finite memory suboptimal design for partially observed markov decision processes we develop bounds on the value function and a suboptimal design for the partially observed markov decision process these bounds and suboptimal design are based on the m most recent observations and actions an a priori measure of the quality of these bounds is given we show that larger m implies tighter bounds an operations count analysis indicates that a z m 1 s multiplications and additions are required per successive approximations iteration of the suboptimal design algorithm where a z and s are the action observation and state spaces respectively suggesting the algorithm is of potential use for problems with large state spaces a preliminary numerical study indicates that the quality of the suboptimal design can be excellent

planning and acting in partially observable stochastic domains in this paper we bring techniques from operations research to bear on the problem of choosing optimal actions in partially observable stochastic domains we begin by introducing the theory of markov decision processes mdps and partially observable mdps pomdps we then outline a novel algorithm for solving pomdps off line and show how in some cases a finite memory controller can be extracted from the solution to a pomdp we conclude with a discussion of how our approach relates to previous work the complexity of finding exact solutions to pomdps and of some possibilities for finding approximate solutions

a two state partially observable markov decision process with three actions abstract a process can be in either a stable or an unstable state interchangeably the true state is unobservable and can only be inferred from observations three actions are available continue with the process con repair the process for a certain fee bring the process to the stable state rep and obtain the state of the process for a cost ins the objective is to maximize the expected discounted value of the total future profits we formulate the problem as a discrete time partially observable markov decision process pomdp we show that the expected profit function is convex and strictly increasing and that the optimal policy has either one or two control limits also we show that dominance in expectation the expected revenue is larger in the stable state than in the unstable state suffices for a control limit structure

maximum likelihood estimation in semiparametric selection bias models with application to aids vaccine trials summary the following problem is treated given s possibly selection biased samples from an unknown distribution function and assuming that the sampling rule weight functions for each of the samples are mathematically specified up to a common unknown finite dimensional parameter how can we use the data to estimate the unknown parameters we propose a simple maximum partial likelihood method for deriving the semiparametric maximum likelihood estimator a discussion of assumptions under which the selection bias model is identifiable and uniquely estimable is presented we motivate the need for the methodology by discussing the generalised logistic regression model gilbert self ashby 1998 a semiparametric selection bias model which is useful for assessing from vaccine trial data how the efficacy of an hiv vaccine varies with characteristics of the exposing virus we show through simulations and an example that the maximum likelihood estimator in the generalised logistic regression model has satisfactory finite sample properties

a nonlinear programming model for partially observable markov decision processes finite horizon case the concept of partially observable markov decision processes was born to handle the problem of lack of information about the state of a markov decision process if the state of the system is unknown to the decision maker then an obvious approach is to gather information that is helpful in selecting an action this problem was already solved using the theory of markov processes we construct a nonlinear programming model for the same problem and develop a solution algorithm that turns out to be a policy iteration algorithm the policies found this way are easier to use than the policies found by the existing method although they have the same optimal objective value

on the structure of optimal real time encoders and decoders in noisy communication the output of a discrete time markov source must be encoded into a sequence of discrete variables the encoded sequence is transmitted through a noisy channel to a receiver that must attempt to reproduce reliably the source sequence encoding and decoding must be done in real time and the distortion measure does not tolerate delays the structure of real time encoding and decoding strategies that jointly minimize an average distortion measure over a finite horizon is determined the results are extended to the real time broadcast problem and a real time variation of the wyner ziv problem

solution procedures for partially observed markov decision processes we present three algorithms to solve the infinite horizon expected discounted total reward partially observed markov decision process pomdp each algorithm integrates a successive approximations algorithm for the pomdp due to a smallwood and e sondik with an appropriately generalized numerical technique that has been shown to reduce cpu time until convergence for the completely observed case the first technique is reward revision the second technique is reward revision integrated with modified policy iteration the third is a standard extrapolation a numerical study indicates the potentially significant computational value of these algorithms

applied smoothing techniques for data analysis the kernel approach with s plus illustrations 1 density estimation for exploring data 2 density estimation for inference 3 nonparametric regression for exploring data 4 inference with nonparametric regression 5 checking parametric regression models 6 comparing regression curves and surfaces 7 time series data 8 an introduction to semiparametric and additive models references

optimal design of sequential real time communication systems optimal design of sequential real time communication of a markov source over a noisy channel is investigated in such a system the delay between the source output and its reconstruction at the receiver should equal a fixed prespecified amount an optimal communication strategy must minimize the total expected symbol by symbol distortion between the source output and its reconstruction design techniques or performance bounds for such real time communication systems are unknown in this paper a systematic methodology based on the concepts of information structures and information states to search for an optimal real time communication strategy is presented this methodology trades off complexity in communication length linear in contrast to doubly exponential with complexity in alphabet sizes doubly exponential in contrast to exponential as the communication length is usually order of magnitudes bigger than the alphabet sizes the proposed methodology simplifies the search for an optimal communication strategy in spite of this simplification the resultant optimality equations cannot be solved efficiently using existing algorithmic techniques the main idea is to formulate a zero delay communication problem as a dynamic team with nonclassical information structure then an appropriate choice of information states converts the dynamic team problem into a centralized stochastic control problem in function space thereafter markov decision theory is used to derive nested optimality equations for choosing an optimal design for infinite horizon problems these optimality equations give rise to a fixed point functional equation communication systems with fixed finite delay constraint a higher order markov source and channels with memory are treated in the same manner after an appropriate expansion of the state space thus this paper presents a comprehensive methodology to study different variations of real time communication

longitudinal data analysis using generalized linear models summary this paper proposes an extension of generalized linear models to the analysis of longitudinal data we introduce a class of estimating equations that give consistent estimates of the regression parameters and of their variance under mild assumptions about the time dependence the estimating equations are derived without specifying the joint distribution of a subject s observations yet they reduce to the score equations for multivariate gaussian outcomes asymptotic theory is presented for the general class of estimators specific cases in which we assume independence m dependence and exchangeable correlation structures from each subject are discussed efficiency of the proposed estimators in two simple situations is considered the approach is closely related to quasi likelih ood some key ironh estimating equation generalized linear model longitudinal data quasi likelihood repeated measures

nuisance parameter elimination for proportional likelihood ratio models with nonignorable missingness and random truncation we show that the proportional likelihood ratio model proposed recently by luo tsai 2012 enjoys model invariant properties under certain forms of nonignorable missing mechanisms and randomly double truncated data so that target parameters in the population can be estimated consistently from those biased samples we also construct an alternative estimator for the target parameters by maximizing a pseudo likelihood that eliminates a functional nuisance parameter in the model the corresponding estimating equation has a u statistic structure as an added advantage of the proposed method a simple score type test is developed to test a null hypothesis on the regression coefficients simulations show that the proposed estimator has a small sample efficiency similar to that of the nonparametric likelihood estimator and performs well for certain nonignorable missing data problems

sunoptimal policies with bounds for parameter adaptive decision processes a parameter adaptive decision process is a sequential decision process where some parameter or parameter set impacting the rewards and or transitions of the process is not known with certainty signals from the performance of the system can be processed by the decision maker as time progresses yielding information regarding which parameter set is operative active learning is an essential feature of these processes and the decision maker must choose actions that simultaneously guide the system in a preferred direction as well as yield information that can be used to better prescribe future actions if the operative parameter set is known with certainty the parameter adaptive problem reduces to a conventional stochastic dynamic program which is presumed solvable previous authors have shown how to use these solutions to generate suboptimal policies with performance bounds for the parameter adaptive problem here it is shown that some desirable characteristics of those bounds are shared by a larger cla