asymptotically efficient adaptive allocation schemes for controlled markov chains finite parameter space the authors consider a controlled markov chain whose transition probabilities and initial distribution are parametrized by an unknown parameter theta belonging to some known parameter space theta there is a one step reward associated with each pair of control and the following state of the process the objective is to maximize the expected value of the sum of one step rewards over an infinite horizon the loss associated with a control scheme at a parameter value is the function of time giving the difference between the maximum reward that could have been achieved if the parameter were known and the reward achieved by the scheme since it is impossible to minimize the loss uniformly for all parameter values the authors define uniformly good adaptive control schemes and restrict attention to these schemes they develop a lower bound on the loss associated with any uniformly good control scheme they construct an adaptive control scheme whose loss equals the lower bound for every parameter value and is therefore asymptotically efficient

machine learning and nonparametric bandit theory in its most basic form bandit theory is concerned with the design problem of sequentially choosing members from a given collection of random variables so that the regret i e r sub n spl sigma sub j spl mu spl mu sub j et sub n j grows as slowly as possible with increasing n here spl mu sub j is the expected value of the bandit arm i e random variable indexed by j t sub n j is the number of times arm j has been selected in the first n decision stages and spl mu sup sup sub j spl mu sub j the present paper contributes to the theory by considering the situation in which observations are dependent to begin with the dependency is presumed to depend only on past observations of the same arm but later we allow that it may be with respect to the entire past and that the set of arms is infinite this brings queues and more generally controlled markov processes into our purview thus our black box methodology is suitable for the case when the only observables are cost values and in particular the probability structure and loss function are unknown to the designer the conclusion of the analysis is that under lenient conditions using algorithms prescribed herein risk growth is commensurate with that in the simplest i i d cases our methods represent an alternative to stochastic approximation perturbation analysis ideas for tuning queues

existence of solutions for third order boundary value problems on a time scale abstract let t be a closed subset of r with inf t and sup t for the third order nonlinear equation u 3 f t u u u t t where f t r3 r is continous we assume solutions of initial value problems are unique and extend to t we consider questions of the uniqueness of solutions implying the existence of solutions for conjugate boundary problems on t

solving singular second order three point boundary value problems using reproducing kernel hilbert space method this paper investigates the numerical solutions of singular second order three point boundary value problems using reproducing kernel hilbert space method it is a relatively new analytical technique the solution obtained by using the method takes the form of a convergent series with easily computable components however the reproducing kernel hilbert space method cannot be used directly to solve a singular second order three point boundary value problem so we convert it into an equivalent integro differential equation which can be solved using reproducing kernel hilbert space method four numerical examples are given to demonstrate the efficiency of the present method the numerical results demonstrate that the method is quite accurate and efficient for singular second order three point boundary value problems

adaptive control of constrained finite markov chains an adaptive control algorithm is presented for constrained finite controlled markov chains with unknown transition probabilities a finite set of algebraic constraints has been considered the lagrange multipliers approach is used to solve this constrained optimization problem this scheme is such that at each time n estimates the control policy on the basis on bush mosteller scheme which is related to stochastic approximation procedures we present the asymptotic properties convergence and order of convergence rate of the algorithm they follow from the law of dependent large numbers martingales theory and lyapunov function analysis approaches

boundary value problems for second order nonlinear differential equations on infinite intervals in this paper we consider boundary value problems for nonlinear differential equations on the semi axis 0 and also on the whole axis under the assumption that the left hand side being a second order linear differential expression belongs to the weyl limit circle case the boundary value problems are considered in the hilbert spaces l2 0 and l2 and include boundary conditions at infinity the existence and uniqueness results for solutions of the considered boundary value problems are established

multiple solutions of singular three point boundary value problems on 0 abstract in this paper we study the existence of single and multiple solutions of three point boundary value problems for the following nonlinear singular second order differential equations x t p x t q x t h t f t x t 0 t 0 a x 0 b x 0 k x c 0 lim t x t e r t d 0 where p b 0 a k 0 q 0 0 r 0 p p 2 4 q 2 h 0 0 is continuous and may be singular at t 0 f 0 0 is continuous and may take a negative value by applying the technique of lower and upper solutions and the theory of topological degree we obtain the conditions for the existence of at least one solution and at least three solutions respectively

solvability of singular second order m point boundary value problems abstract let f 0 1 r 2 r be a function satisfying caratheodory s conditions and 1 t e t l 1 0 1 let i 0 1 a i r i 1 m 2 0 1 2 m 2 1 be given this paper is concerned with the problem of existence of a c 1 0 1 solution for the m point boundary value problem x f t x t x t e t 0 t 1 0 1 x 0 0 x 1 i 1 m 2 a i x i the proof of our main result is based upon the leray schauder continuation theorem

a survey of some results in stochastic adaptive control some results in discrete time stochastic adaptive control are surveyed the survey divides itself into two parts bayesian and non bayesian adaptive control in the former area the problems of converting an incompletely observed system into a completely observed one multi armed bandit processes bayesian adaptive control of markov chains and bayesian adaptive control of linear systems are exposed and surveyed in the latter area non bayesian adaptive control of markov chains and the self tuning regulator are dealt with proofs are given where appropriate to illustrate the methods involved

multiple positive solutions for first order nonlinear integro differential equations in banach spaces by using the fixed point index theory the author obtains the existence of two positive solutions for a boundary value problem of first order nonlinear integro differential equations on an infinite interval in a real banach space

learning to optimize via information directed sampling we propose information directed sampling a new approach to online optimization problems in which a decision maker must balance between exploration and exploitation while learning from partial feedback each action is sampled in a manner that minimizes the ratio between squared expected single period regret and a measure of information gain the mutual information between the optimal action and the next observation we establish an expected regret bound for information directed sampling that applies across a very general class of models and scales with the entropy of the optimal action distribution we illustrate through simple analytic examples how information directed sampling accounts for kinds of information that alternative approaches do not adequately address and that this can lead to dramatic performance gains for the widely studied bernoulli gaussian and linear bandit problems we demonstrate state of the art simulation performance the electronic companion is available at https doi org 10 128

necessary and sufficient conditions for the existence of positive solutions of fourth order multi point boundary value problems abstract this paper investigates the existence of positive solutions of singular multi point boundary value problems of fourth order ordinary differential equation with p laplacian a necessary and sufficient condition for the existence of c 2 0 1 positive solution as well as pseudo c 3 0 1 positive solution is given by means of the fixed point theorems on cones

positive solutions to singular multi point dynamic eigenvalue problems with mixed derivatives abstract this paper considers a singular m point dynamic eigenvalue problem on time scales t p t u t f t u t t 0 1 t u 0 i 1 m 2 a i u i u 1 p 1 u 1 i 1 m 2 b i p i u i we allow f t w to be singular at w 0 and t 0 by constructing the green s function and studying its positivity eigenvalue intervals in which there exist positive solutions of the above problem are obtained by making use of the fixed point index theory

optimal strategies for a class of sequential control problems with precedence relations consider the following multi phase project management problem each project is divided into several phases all projects enter the next phase at the same point chosen by the decision maker based on observations up to that point within each phase one can pursue the projects in any order when pursuing the project with one unit of resource the project state changes according to a markov chain the probability distribution of the markov chain is known up to an unknown parameter when pursued the project generates a random reward depending on the phase and the state of the project and the unknown parameter the decision maker faces two problems a how to allocate resources to projects within each phase and b when to enter the next phase so that the total expected reward is as large as possible in this paper we formulate the preceding problem as a stochastic scheduling problem and propose asymptotic optimal strategies which minimize the shortfall from perfect information payoff concrete examples are given to illustrate our method

a reduce and solve approach for the multiple choice multidimensional knapsack problem abstract the multiple choice multidimensional knapsack problem mmkp is a well known np hard combinatorial optimization problem with a number of important applications in this paper we present a reduce and solve heuristic approach which combines problem reduction techniques with an integer linear programming ilp solver cplex the key ingredient of the proposed approach is a set of group fixing and variable fixing rules these fixing rules rely mainly on information from the linear relaxation of the given problem and aim to generate reduced critical subproblem to be solved by the ilp solver additional strategies are used to explore the space of the reduced problems extensive experimental studies over two sets of 37 mmkp benchmark instances in the literature show that our approach competes favorably with the most recent state of the art algorithms in particular for the set of 27 conventional benchmarks the proposed approach finds an improved best lower bound for 11 instances and as a by product improves all the previous best upper bounds for the 10 additional instances with irregular structures the method improves 7 best known results

asymptotically efficient adaptive choice of control laws incontrolled markov chains we consider a controlled markov chain on a general state space whose transition probabilities are parameterized by an unknown parameter belonging to a compact metric space there is a one step reward associated with each pair of control and the following state of the process given a finite set of stationary control laws under each of which the markov chain is uniformly recurrent an optimal control law is this set is one that maximizes the long run average reward in ignorance of the parameter value we construct an adaptive control rule which uses the optimal control law s at a relative frequency of 1 o n 1 log n and show that this relative frequency gives an asymptotically optimal balance between the control objective and the amount of information needed to learn about the unknown parameter the basic idea underlying this construction is to introduce suitable uncertainty adjustments via sequential testing theory into the certainty equivalence rule thus resolving the apparent dilemma between control and information

certainty equivalence control with forcing revisited abstract certainty equivalence control with forcing has been shown to be optimal for several stochastic adaptive control problems with the average cost per unit time criterion recently researchers have started looking at stochastic adaptive control problems with a view to minimizing the rate of increase of the learning loss this criterion is stronger than the average cost per unit time criterion certainty equivalence control with forcing does not usually suffice for the learning loss criterion and one has to develop fairly complicated schemes in order to achieve optimality the objective of this paper is to see how well one might be able to do with a certainty equivalence control with forcing type of scheme in particular we construct a class of such schemes whose learning loss is o log n 1 for 0 whereas optimal schemes typically have a o log n learning loss

multiple positive solutions of the singular boundary value problems for second order differential equations on the half line abstract in this paper the existence of multiple positive solutions for singular sturm liouville boundary value problems on the half line is investigated by applying the fixed point index theorem of cone map some existence and multiplicity results of positive solutions are derived our results improve and generalize many well known results two examples are presented to demonstrate the application of our main results

saddlepoint expansions for sums of markov dependent variables on a continuous state space based on the conjugate kernel studied in iscoe et al 1985 we derive saddlepoint expansions for either the density or distribution function of a sumf x1 f x n where thex i s constitute a markov chain the chain is assumed to satisfy a strong recurrence condition which makes the results here very similar to the classical results for i i d variables in particular we establish also conditions under which the expansions hold uniformly over the range of the saddlepoint expansions are also derived for sums of the formf x1 x0 f x2 x1 f x n xn 1 although the uniformity result just mentioned does not generalize

positive solutions of singular sublinear emden fowler boundary value problems abstract this paper is about the dirichlet boundary value problem of singular sublinear emden fowler equations a necessary and sufficient condition for the existence of positive solutions to this problem has been obtained by using the method of lower and upper solutions the uniqueness of positive solutions has also been discussed

asymptotically efficient adaptive allocation schemes for controlled i i d processes finite parameter space the authors consider a controlled i i d independently identically distributed process whose distribution is parametrized by an unknown parameter theta belonging to some known parameter space theta and a one step reward associated with each pair of control and the following state of the process the objective is to maximize the expected value of the sum of one step rewards over an infinite horizon by introducing the loss associated with a control scheme it is shown that the problem is equivalent to minimizing this loss uniformly good adaptive control schemes are defined and emphasized a lower bound on the loss associated with any uniformly good control scheme is developed finally an adaptive control scheme is constructed whose loss equals the lower bound and is therefore asymptotically efficient

some new results for singular problems with sign changing nonlinearities we present some general existence theorems for second order boundary value problems with sign changing nonlinearities both the upper and lower solution approach and the growth restriction approach are discussed

existence of positive solutions for sturm liouville boundary value problems on the half line in this paper we establish sufficient conditions to guarantee the existence of at least one positive solution a unique positive solution and multiple positive solutions for the sturm liouville boundary value problem on the half line by using an effective operator the fixed point theorems in cone especially krasnoselskii fixed point theorem can be applied to such systems and then existence criteria are established the interesting point of the results is that the nonlinear term f can be sign changing

the existence of multiple positive solutions to boundary value problems of nonlinear delay differential equations with countably many singularities on infinite interval in this paper we consider the existence of countably many positive solutions to a boundary value problem of a nonlinear delay differential equation with countably many singularities on infinite interval f x t a t f t x t x t 0 0 x t 0 where f r r is an increasing homeomorphism and a positive homomorphism with f 0 0 x t is a function in c r 0 r defined by x t s x t s for r s 0 and x c r 0 r by using the fixed point index theory and a new fixed point theorem in a cone we provide sufficient conditions for the existence of multiple positive solutions to the above boundary value problem the conclusions in this paper essentially extend and improve the known results

the upper and lower solution method for a class of singular nonlinear second order three point boundary value problems in this paper we develop the upper and lower solution method and the monotone iterative technique for a class of singular nonlinear second order three point boundary value problems a maximum principle is established and some new existence results are obtained

third order boundary value problems with sign changing solutions we consider the third order nonlinear differential equation u t f t u t u t u t a e t 0 1 satisfying u 0 u 0 u 1 0 and u 0 u 1 u 1 0 where f 0 1 r3 r is lp caratheodory 1 p 1 and p 1

existence of positive solutions for functional differential equations under suitable conditions on f t yt the boundary value problem of second order functional differential equation fde with the form fde y t f t yt 0 for t 0 1 a bc y t y t t for t 0 bvp y t y e t for t 1 1 a has at least one positive solution moreover we also apply this main result to establish several existence theorems which guarantee bvp has the multiple positive solutions

on numerical studies of multi point boundary value problem and its fold bifurcation based upon shooting method we set up a numerical method for approximating solutions and fold bifurcation solutions of second order multi point boundary value problems we prove the equivalence of the regularity of solutions and simple fold bifurcation solutions between the numerical method and the multi point boundary value problems many numerical experiments are carried out to illustrate the theoretical analysis from which we explore that the fold bifurcation phenomenon are generic in multi point boundary value problems although there is no theoretical analysis in the literature so far

asymptotically efficient adaptive allocation rules for the multiarmed bandit problem with switching cost the authors consider multiarmed bandit problems with switching cost define uniformly good allocation rules and restrict attention to such rules they present a lower bound on the asymptotic performance of uniformly good allocation rules and construct an allocation scheme that achieves the bound it is found that despite the inclusion of a switching cost the proposed allocation scheme achieves the same asymptotic performance as the optimal rule for the bandit problem without switching cost this is made possible by grouping together samples into blocks of increasing sizes thereby reducing the number of switches to o log n finally an optimal allocation scheme for a large class of distributions which includes members of the exponential family is illustrated

green s function for a third order generalized right focal problem abstract we determine the green s function for the third order three point generalized right focal boundary value problem x t 0 t 1 t t 3 x t 1 x t 2 0 x t 3 x t 3 0 where 0 and determine conditions on the coefficients and boundary points to ensure its positivity then we use the krasnoselskii and leggett and williams fixed point theorems to prove the existence of solutions to the nonlinear problem x t f t x t t 1 t t 3 x t 1 x t 2 0 x t 3 x t 3 0 for certain constraints placed on nonnegative f

existence and uniqueness for an m point boundary value problem at resonance on infinite intervals this paper investigates an m point boundary value problem with nonlinear boundary conditions at resonance on infinite intervals by establishing a general continuation theorem the existence and uniqueness of unbounded solutions is obtained on the basis of a suitable homotopy leray schauder degree properties and a priori estimate this paper provides a new method to solve boundary value problems with nonlinear boundary conditions at resonance on infinite intervals

boundary value problems for singular second order functional differential equations abstract we consider general linear boundary value problems for equations of the form y f x y x 0 0 x x is continuous and f x y has a singularity at y 0 the results improve and extend earlier results for the case x x due to taliaferro 1979 and gatica 1989

an adaptive sampling algorithm for solving markov decision processes based on recent results for multiarmed bandit problems we propose an adaptive sampling algorithm that approximates the optimal value of a finite horizon markov decision process mdp with finite state and action spaces the algorithm adaptively chooses which action to sample as the sampling process proceeds and generates an asymptotically unbiased estimator whose bias is bounded by a quantity that converges to zero at rate lnn n wheren is the total number of samples that are used per state sampled in each stage the worst case running time complexity of the algorithm iso a n h independent of the size of the state space where a is the size of the action space andh is the horizon length the algorithm can be used to create an approximate receding horizon control to solve infinite horizon mdps to illustrate the algorithm computational results are reported on simple examples from inventory control

a singular non local problem at resonance abstract we study a second order non local problem using the coincidence degree theory we show the existence of a solution whose derivative is singular at the right end point of the interval which is a new result for any resonant non local problem under the non local condition we find a general way to ensure that the differential operator is a fredholm operator of index zero

existence of nontrivial solutions for a nonlinear sturm liouville problem with integral boundary conditions abstract this paper deals with the existence of nontrivial solutions for a nonlinear singular sturm liouville problem with integral boundary conditions

solvability of an m point boundary value problem for second order ordinary differential equations let 0 1 r2 r be a function satisfying caratheodory s conditions and e t l1 0 1 let i 0 1 ai r all of the ai s having the same sign i 1 2 m 2 0 1 2 m 2 1 be given this paper is concerned with the problem of existence of a solution for the m point boundary value problem x t t x t x t e t t 0 1 e formula conditions for the existence of a solution for the above boundary value problem are given using the leray schauder continuation theorem

positive solutions for three point boundary value problem on the half line in this paper we consider the existence of positive solutions for the following boundary value problem on the half line r t x t f t x t x t 0 email protected 0 x 0 email protected x limt x t 0 by applying fixed point theorems we obtain a variety of existence results in particular the nonlinear term is involved with the first order derivative

on positive solutions of boundary value problems on the half line we study the existence of positive solutions of boundary value problems on the half line for differential equations of second order the krasnoselskii fixed point theorem on cone compression and expansion is used

an iterative pseudo gap enumeration approach for the multidimensional multiple choice knapsack problem abstract the multidimensional multiple choice knapsack problem mmkp is an important np hard combinatorial optimization problem with many applications we propose a new iterative pseudo gap enumeration approach to solving mmkps the core of our algorithm is a family of additional cuts derived from the reduced costs constraint of the nonbasic variables by reference to a pseudo gap we then introduce a strategy to enumerate the pseudo gap values joint with cplex we evaluate our approach on two sets of benchmark instances and compare our results with the best solutions reported by other heuristics in the literature it discovers 10 new better lower bounds on 37 well known benchmark instances with a time limit of 1 hour for each instance we further give direct comparison between our algorithm and one state of the art reduce and solve approach on the same machine with the same cplex experimental results show that our algorithm is very competitive outperforming reduce and solve on 18 cases out of 37

existence of three positive solutions for m point boundary value problems on infinite intervals abstract in this paper by using the avery peterson fixed point theorem on a cone we establish the existence of three positive solutions for m point boundary value problems on infinite intervals p x t t f t x t x t 0 0 t x 0 i 1 m 2 i x i lim t x t 0 where p s s p 2 s p 1 r r f t u v r 3 r is a continuous function r 0 i 0 and 0 1 2 m 2 are given

infinite boundary value problems for nth order nonlinear impulsive integro differential equations in banach spaces abstract in this paper the existence of unbounded nonnegative solutions of a boundary value problem for n th order impulsive integro differential equations on an infinite interval in banach spaces is obtained by means of the monch fixed point theory in a banach space

on improper integrals and differential equations in ordered banach spaces in this paper we study the existence of improper integrals of vector valued mappings the so obtained results combined with fixed point results in partially ordered functions spaces are then applied to derive existence and comparison results for least and greatest solutions of initial and boundary value problems in ordered banach spaces the considered problems can be singular functional nonlocal implicit and discontinuous concrete examples are also solved

second order boundary value problems on an unbounded domain abstract we consider the second order nonlinear differential equation p t u t f t u t u t a e in 0 satisfying two sets of boundary conditions u 0 0 lim t u t 0 and u 0 0 lim t u t 0 where f 0 r 2 r is caratheodory with respect to l 1 0 p c 0 c 1 0 and p t 0 for all t 0 we obtain the existence of at least one solution using the leray schauder continuation principle

positive solutions for singular non linear third order periodic boundary value problems abstract in this paper we are concerned with the problem of the existence of positive solutions for non linear third order periodic boundary value problem u 3 u f t u 0 t 2 u i 0 u i 2 i 0 1 2 here 0 1 3 is a positive constant and our non linearity f t u may be singular at u 0 the proof relies on a non linear alternative of leray schauder type and on krasnoselskii fixed point theorem on compression and expansion of cones

constant sign solutions for a system of third order generalized right focal problems abstract we consider the following system of third order three point generalized right focal boundary value problems u i t f i t u 1 t u 2 t u n t t a b u i a u i t i 0 i u i b i u i b 0 where i 1 2 n i 0 and i 0 using a nonlinear alternative of leray schauder type and krasnosel skii s fixed point theorem we establish the existence of one or more constant sign solutions for the system

positive solutions for multi point boundary value problem on the half line this paper presents a variety of existence results for nonlinear multi point boundary value problem on the half line in particular we consider x t px t qx t f t x t 0 t 0 x 0 x 0 i 1nkix i a limt x t ert b r 0 p p2 4q2 existence results are established using fixed point theorems on cone

a dirichelet neumann m point bvp with a p laplacian like operator abstract let be odd increasing homeomorphisms from r onto r satisfying 0 0 0 and let f a b r r r be a function satisfying carath e odory s conditions let i r i a b i 1 m 2 a 1 2 m 2 b be given we are interested in the problem of existence of solutions for the m point boundary value problem u f t u u t a b u a 0 u b i 1 m 2 i u i in the resonance and non resonance cases we say that this problem is at resonance if the associated problem u 0 t a b u a 0 u b i 1 m 2 i u i has a non trivial solutions this is the case if and only if i 1 m 2 i 1 our results use topological degree methods interestingly enough in the non resonance case i e when i 1 m 2 i 1 the sign of degree for the relevant operator depends on whether i 1 m 2 i 1 or i 1 m 2 i 1

on positive solutions of boundary value problems for second order functional differential equations on infinite intervals we study the existence of positive solutions for the following boundary value problem on infinite interval for second order functional differential equations x t px t qx t f t xt 0 t 0 x t x t t t 0 limt x t 0 and x t px t qx t f t xt x t 0 t 0 x t x t t t 0 limt x t 0 where p 0 2 2 0 and q 0 the fixed point theorem on cone is used

budgeting with bounded multiple choice constraints abstract we consider a budgeting problem where a specified number of projects from some disjoint classes has to be selected such that the overall gain is largest possible and such that the costs of the chosen projects do not exceed a fixed upper limit the problem has several application in government budgeting planning and as relaxation from other combinatorial problems it is demonstrated that the problem can be transformed to an equivalent multiple choice knapsack problem through dynamic programming a naive transformation however leads to a drastic increase in the number of variables thus we propose an algorithm for the continuous problem based on dantzig wolfe decomposition a master problem solves a continuous multiple choice knapsack problem knowing only some extreme points in each of the transformed classes the individual subproblems find extreme points for each given direction using a median search algorithm an integer optimal solution is then derived by using the dynamic programming transformation to a multiple choice knapsack problem for an expanding core the individual classes are considered in an order given by their gradients and the transformation to a multiple choice knapsack problem is performed when needed in this way only a dozen of classes need to be transformed for standard instances from the literature computational experiments are presented showing that the developed algorithm is orders of magnitude faster than a general lp mip algorithm

positive solutions for singular semipositone boundary value problems on infinite intervals by using the fixed point theory on a cone with a special norm and space we discuss the existence of positive solutions for a class of semipositone boundary value problems on infinite intervals the work improves many known results including singular and non singular cases

symmetric positive solutions for nonlocal boundary value problems of fourth order abstract the singular fourth order nonlocal boundary value problem u t h t f t u 0 t 1 u 0 u 1 0 1 p s u s d s u 0 u 1 0 1 q s u s d s is considered under some suitable conditions concerning the first eigenvalue of the corresponding linear operator where p q l 1 0 1 h 0 1 0 is continuous symmetric on 0 1 and may be singular at t 0 and t 1 f 0 1 0 0 is continuous and f x is symmetric on 0 1 for all x 0 the existence of at least one symmetric positive solution is obtained by the application of the fixed point index in cones

optimal adaptive policies for markov decision processes in this paper we consider the problem of adaptive control for markov decision processes we give the explicit form for a class of adaptive policies that possess optimal increase rate properties for the total expected finite horizon reward under sufficient assumptions of finite state action spaces and irreducibility of the transition law a main feature of the proposed policies is that the choice of actions at each state and time period is based on indices that are inflations of the right hand side of the estimated average reward optimality equations