technical note a note on relaxations of the choice network revenue management dynamic program in recent years several approximation methods have been proposed for the choice network revenue management problem these approximation methods are proposed because the dynamic programming formulation of the choice network revenue management problem is intractable even for moderately sized instances in this paper we consider three approximation methods that obtain upper bounds on the value function namely the choice deterministic linear program cdlp the affine approximation af and the piecewise linear approximation pl it is known that the piecewise linear approximation bound is tighter than the affine bound which in turn is tighter than cdlp in this paper we prove bounds on how much the affine and piecewise linear approximations can tighten cdlp we show i the gap between the af and cdlp bounds is at most a factor of 1 1 mini ri1 where ri1 0 are the resource capacities and ii the gap between the piecewise linear and cdlp bounds is within a factor of 2 moreover we show that thes

finite sample analysis of least squares policy iteration in this paper we report a performance bound for the widely used least squares policy iteration lspi algorithm we first consider the problem of policy evaluation in reinforcement learning that is learning the value function of a fixed policy using the least squares temporal difference lstd learning method and report finite sample analysis for this algorithm to do so we first derive a bound on the performance of the lstd solution evaluated at the states generated by the markov chain and used by the algorithm to learn an estimate of the value function this result is general in the sense that no assumption is made on the existence of a stationary distribution for the markov chain we then derive generalization bounds in the case when the markov chain possesses a stationary distribution and is b mixing finally we analyze how the error at each policy evaluation step is propagated through the iterations of a policy iteration method and derive a performance bound for the lspi algorithm

strong semismoothness of eigenvalues of symmetric matrices and its application to inverse eigenvalue problems it is well known that the eigenvalues of a real symmetric matrix are not everywhere differentiable a classical result of ky fan states that each eigenvalue of a symmetric matrix is the difference of two convex functions which implies that the eigenvalues are semismooth functions based on a recent result of the authors it is further proved in this paper that the eigenvalues of a symmetric matrix are strongly semismooth everywhere as an application it is demonstrated how this result can be used to analyze the quadratic convergence of newton s method for solving inverse eigenvalue problems ieps and generalized ieps with multiple eigenvalues

practical solution techniques for first order mdps many traditional solution approaches to relationally specified decision theoretic planning problems e g those stated in the probabilistic planning domain description language or ppddl ground the specification with respect to a specific instantiation of domain objects and apply a solution approach directly to the resulting ground markov decision process mdp unfortunately the space and time complexity of these grounded solution approaches are polynomial in the number of domain objects and exponential in the predicate arity and the number of nested quantifiers in the relational problem specification an alternative to grounding a relational planning problem is to tackle the problem directly at the relational level in this article we propose one such approach that translates an expressive subset of the ppddl representation to a first order mdp fomdp specification and then derives a domain independent policy without grounding at any intermediate step however such generality does not come without its own set of challenges the purpose of this article is to explore practical solution techniques for solving fomdps to demonstrate the applicability of our techniques we present proof of concept results of our first order approximate linear programming foalp planner on problems from the probabilistic track of the icaps 2004 and 2006 international planning competitions

acceleration operators in the value iteration algorithms for markov decision processes we study the general approach to accelerating the convergence of the most widely used solution method of markov decision processes mdps with the total expected discounted reward inspired by the monotone behavior of the contraction mappings in the feasible set of the linear programming problem equivalent to the mdp we establish a class of operators that can be used in combination with a contraction mapping operator in the standard value iteration algorithm and its variants we then propose two such operators which can be easily implemented as part of the value iteration algorithm and its variants numerical studies show that the computational savings can be significant especially when the discount factor approaches one and the transition probability matrix becomes dense in which the standard value iteration algorithm and its variants suffer from slow convergence

numerical solution of continuous state dynamic programs using linear and spline interpolation this paper demonstrates that the computational effort required to develop numerical solutions to continuous state dynamic programs can be reduced significantly when cubic piecewise polynomial functions rather than tensor product linear interpolants are used to approximate the value function tensor product cubic splines represented in either piecewise polynomial or b spline form and multivariate hermite polynomials are considered computational savings are possible because of the improved accuracy of higher order functions and because the smoothness of higher order functions allows efficient quasi newton methods to be used to compute optimal decisions the use of the more efficient piecewise polynomial form of the spline was slightly superior to the use of hermite polynomials for the test problem and easier to program in comparison to linear interpolation use of splines in piecewise polynomial form reduced the cpu time to obtain results of equivalent accuracy by a factor of 250 330 for a stochastic

multiple eigenvalue sensitivity analysis abstract this paper surveys the development of the sensitivity analysis of multiple eigenvalues and the associated invariant subspaces of two kinds of matrix eigenvalue problems in which the matrices are analytically dependent on several parameters

a polyhedral approach to online bipartite matching we study the i i d online bipartite matching problem a dynamic version of the classical model where one side of the bipartition is fixed and known in advance while nodes from the other side appear one at a time as i i d realizations of a uniform distribution and must immediately be matched or discarded we consider various relaxations of the polyhedral set of achievable matching probabilities introduce valid inequalities and discuss when they are facet defining we also show how several of these relaxations correspond to ranking policies and their time dependent generalizations we finally present a computational study of these relaxations and policies to determine their empirical performance

relaxations of weakly coupled stochastic dynamic programs we consider a broad class of stochastic dynamic programming problems that are amenable to relaxation via decomposition these problems comprise multiple subproblems that are independent of each other except for a collection of coupling constraints on the action space we fit an additively separable value function approximation using two techniques namely lagrangian relaxation and the linear programming lp approach to approximate dynamic programming we prove various results comparing the relaxations to each other and to the optimal problem value we also provide a column generation algorithm for solving the lp based relaxation to any desired optimality tolerance and we report on numerical experiments on bandit like problems our results provide insight into the complexity versus quality trade off when choosing which of these relaxations to implement

a proof of stability conjecture for three dimensional flows we give a proof of the c1 stability conjecture for three dimensional flows i e prove that there exists a hyperbolic structure over the q set for the structurally stable three dimensional flows mafn s proof for the discrete case motivates our proof and we find his perturbation techniques crucial in proving this conjecture we have overcome several new difficulties e g the change of period after perturbation the ergodic closing lemma for flows the existence of dominated splitting over q y where 3 is the set of singularities for the flow the discontinuity of the contracting rate function on singularities etc based on these we finally succeed in separating the singularities from the other periodic orbits for the structurally stable systems i e we create unstable saddle connections if there are accumulations of periodic orbits on the singularities

pathwise optimization for optimal stopping problems we introduce the pathwise optimization po method a new convex optimization procedure to produce upper and lower bounds on the optimal value the price of a high dimensional optimal stopping problem the po method builds on a dual characterization of optimal stopping problems as optimization problems over the space of martingales which we dub the martingale duality approach we demonstrate via numerical experiments that the po method produces upper bounds of a quality comparable with state of the art approaches but in a fraction of the time required for those approaches as a by product it yields lower bounds and suboptimal exercise policies that are substantially superior to those produced by state of the art methods the po method thus constitutes a practical and desirable approach to high dimensional pricing problems furthermore we develop an approximation theory relevant to martingale duality approaches in general and the po method in particular our analysis provides a guarantee on the quality of upper bounds resulting from these approaches and identifies three key determinants of their performance the quality of an input value function approximation the square root of the effective time horizon of the problem and a certain spectral measure of predictability of the underlying markov chain as a corollary to this analysis we develop approximation guarantees specific to the po method finally we view the po method and several approximate dynamic programming methods for high dimensional pricing problems through a common lens and in doing so show that the po method dominates those alternatives this paper was accepted by wei xiong stochastic models and simulation

numerical methods for finding multiple eigenvalues of matrices depending on parameters let a alpha lambda be a square matrix dependent on parameters alpha and lambda of which we choose lambda as the eigenvalue parameter many computational problems are equivalent to finding a point alpha lambda such that a alpha lambda has a multiple eigenvalue lambda lambda at alpha alpha an incomplete qr decomposition of a matrix dependent on several parameters is proposed based on the developed theory two new algorithms are presented for computing multiple eigenvalues of a alpha lambda with geometric multiplicity m ge 2 a third algorithm is designed for the computation of multiple eigenvalues with geometric multiplicity m 1 but which also appears to have local quadratic convergence to semi simple eigenvalues convergence analyses of these methods are given several numerical examples are presented which illustrate the behaviour and applications of our methods

approximating the stochastic knapsack problem the benefit of adaptivity we consider a stochastic variant of the np hard 0 1 knapsack problem in which item values are deterministic and item sizes are independent random variables with known arbitrary distributions items are placed in the knapsack sequentially and the act of placing an item in the knapsack instantiates its size our goal is to compute a solution policy that maximizes the expected value of items successfully placed in the knapsack where the final overflowing item contributes no value we consider both nonadaptive policies that designate a priori a fixed sequence of items to insert and adaptive policies that can make dynamic choices based on the instantiated sizes of items placed in the knapsack thus far an important facet of our work lies in characterizing the benefit of adaptivity for this purpose we advocate the use of a measure called the adaptivity gap the ratio of the expected value obtained by an optimal adaptive policy to that obtained by an optimal nonadaptive policy we bound the adaptivity

approximate dynamic programming for stochastic linear control problems on compact state spaces abstract this paper addresses markov decision processes over compact state and action spaces we investigate the special case of linear dynamics and piecewise linear and convex immediate costs for the average cost criterion this model is very general and covers many interesting examples for instance in inventory management due to the curse of dimensionality the problem is intractable and optimal policies usually cannot be computed not even for instances of moderate size we show the existence of optimal policies and of convex and bounded relative value functions that solve the average cost optimality equation under reasonable and easy to check assumptions based on these insights we propose an approximate relative value iteration algorithm based on piecewise linear convex relative value function approximations besides computing good policies the algorithm also provides lower bounds to the optimal average cost which allow us to bound the optimality gap of any given policy for a given instance the algorithm is applied to the well studied multiple sourcing problem as known from inventory management multiple sourcing is known to be a hard problem and usually tackled by parametric heuristics we analyze several msp instances with two and more suppliers and compare our results to state of the art heuristics for the considered scenarios our policies are always at least as good as the best known heuristic and strictly better in most cases moreover by using the computed lower bounds we show for all instances that the optimality gap has never exceeded 5 and that it has been much smaller for most of them

on constraint sampling in the linear programming approach to approximate dynamic programming in the linear programming approach to approximate dynamic programming one tries to solve a certain linear program the alp that has a relatively small numberk of variables but an intractable numberm of constraints in this paper we study a scheme that samples and imposes a subset ofm

a linearly relaxed approximate linear program for markov decision processes approximate linear programming alp and its variants have been widely applied to markov decision processes mdps with a large number of states a serious limitation of alp is that it has an intractable number of constraints as a result of which constraint approximations are of interest in this paper we define a linearly relaxed approximation linear program lralp that has a tractable number of constraints obtained as positive linear combinations of the original constraints of the alp the main contribution is a novel performance bound for lralp

dynamic bid prices in revenue management we formally derive the standard deterministic linear program lp for bid price control by making an affine functional approximation to the optimal dynamic programming value function this affine functional approximation gives rise to a new lp that yields tighter bounds than the standard lp whereas the standard lp computes static bid prices our lp computes a time trajectory of bid prices we show that there exist dynamic bid prices optimal for the lp that are individually monotone with respect to time we provide a column generation procedure for solving the lp within a desired optimality tolerance and present numerical results on computational and economic performance

approximate dynamic programming via a smoothed linear program we present a novel linear program for the approximation of the dynamic programming cost to go function in high dimensional stochastic control problems lp approaches to approximate dp have typically relied on a natural projection of a well studied linear program for exact dynamic programming such programs restrict attention to approximations that are lower bounds to the optimal cost to go function our program the smoothed approximate linear program is distinct from such approaches and relaxes the restriction to lower bounding approximations in an appropriate fashion while remaining computationally tractable doing so appears to have several advantages first we demonstrate bounds on the quality of approximation to the optimal cost to go function afforded by our approach these bounds are in general no worse than those available for extant lp approaches and for specific problem instances can be shown to be arbitrarily stronger second experiments with our approach on a pair of challenging problems the game of tetris and a queueing network control problem show that the approach outperforms the existing lp approach which has previously been shown to be competitive with several adp algorithms by a substantial margin

shortest paths without a map we study several versions of the shortest path problem when the map is not known in advanced but is specified dynamically we are seeking dynamic decision rules that optimize the worst case ratio of the distance covered to the length of the statically optimal path we describe optimal decision rules for two cases layered graphs of bounded width and two dimensional scenes with unit square obstacles the optimal rules turn out to be intuitive common sense heuristics for slightly more general graphs and scenes we show that no bounded ratio is possible we also show that the computational problem of devising a strategy that achieves a given worst case ratio to the optimum path in a graph is a universal two person game and thus pspace complete whereas optimizing the expected ratio is p hard

a dynamic traveling salesman problem with stochastic arc costs we propose a dynamic traveling salesman problem tsp with stochastic arc costs motivated by applications such as dynamic vehicle routing in which the cost of a decision is known only probabilistically beforehand but is revealed dynamically before the decision is executed we formulate this as a dynamic program dp and compare it to static counterparts to demonstrate the advantage of the dynamic paradigm over an a priori approach we then apply approximate linear programming alp to overcome the dp s curse of dimensionality obtain a semi infinite linear programming lower bound and discuss its tractability we also analyze a rollout version of the price directed policy implied by our alp and derive worst case guarantees for its performance our computational study demonstrates the quality of a heuristically modified rollout policy using a computationally effective a posteriori bound

sufficient conditions for the solvability of an algebraic inverse eigenvalue problem abstract let a a1 a2 an be given n n hermitian matrices and 1 2 n be given real numbers we present new sufficient conditions under which there exist real numbers c1 c2 cn such that the matrix a nt 1 ctat has eigenvalues 1 2 n

a quadratically convergent algorithm for inverse generalized eigenvalue problems abstract in 2018 for inverse generalized symmetric eigenvalue problems a new quadratically convergent algorithm was discovered from simple matrix equations although this algorithm has some nice features compared with the standard newton s method it cannot be applied to multiple eigenvalues in this paper we propose a modified algorithm adapted to an arbitrary set of prescribed eigenvalues moreover we prove its quadratic convergence in a neighborhood of the solutions that satisfy a mild condition

vector fields in the interior of kupka smale systems satisfy axiom a abstract in 1994 s hu conjectured in his paper 1994 trans amer math soc 342 753 772 that axiom a systems are dense in the interior of kupka smale systems we will show that if x is in the interior of kupka smale systems then x satisfies axiom a

optimal toll design a lower bound framework for the asymmetric traveling salesman problem we propose a framework of lower bounds for the asymmetric traveling salesman problem tsp based on approximating the dynamic programming formulation with different basis vector sets we discuss how several well known tsp lower bounds correspond to intuitive basis vector choices and give an economic interpretation wherein the salesman must pay tolls as he travels between cities we then introduce an exact reformulation that generates a family of successively tighter lower bounds all solvable in polynomial time we show that the base member of this family yields a bound greater than or equal to the well known held karp bound obtained by solving the linear programming relaxation of the tsp s integer programming arc based formulation

policy evaluation with temporal differences a survey and comparison policy evaluation is an essential step in most reinforcement learning approaches it yields a value function the quality assessment of states for a given policy which can be used in a policy improvement step since the late 1980s this research area has been dominated by temporal difference td methods due to their data efficiency however core issues such as stability guarantees in the off policy scenario improved sample efficiency and probabilistic treatment of the uncertainty in the estimates have only been tackled recently which has led to a large number of new approaches this paper aims at making these new developments accessible in a concise overview with foci on underlying cost functions the off policy scenario as well as on regularization in high dimensional feature spaces by presenting the first extensive systematic comparative evaluations comparing td lstd lspe fpkf the residual gradient algorithm bellman residual minimization gtd gtd2 and tdc we shed light on the strengths and weaknesses of the methods moreover we present alternative versions of lstd and lspe with drastically improved off policy performance

regularized policy iteration with nonparametric function spaces we study two regularization based approximate policy iteration algorithms namely reg lspi and reg brm to solve reinforcement learning and planning problems in discounted markov decision processes with large state and finite action spaces the core of these algorithms are the regularized extensions of the least squares temporal difference lstd learning and bellman residual minimization brm which are used in the algorithms policy evaluation steps regularization provides a convenient way to control the complexity of the function space to which the estimated value function belongs and as a result enables us to work with rich nonparametric function spaces we derive efficient implementations of our methods when the function space is a reproducing kernel hilbert space we analyze the statistical properties of reg lspi and provide an upper bound on the policy evaluation error and the performance loss of the policy returned by this method our bound shows the dependence of the loss on the number of samples the capacity of the function space and some intrinsic properties of the underlying markov decision process the dependence of the policy evaluation bound on the number of samples is minimax optimal this is the first work that provides such a strong guarantee for a nonparametric approximate policy iteration algorithm

an approximate dynamic programming approach to multidimensional knapsack problems we present an approximate dynamic programming adp approach for the multidimensional knapsack problem mkp we approximate the value function a using parametric and nonparametric methods and busing a base heuristic we propose a new heuristic which adaptively rounds the solution of the linear programming relaxation our computational study suggests athe new heuristic produces high quality solutions fast and robustly bstate of the art commercial packages like cplex require significantly larger computational time to achieve the same quality of solutions c the adp approach using the new heuristic competes successfully with alternative heuristic methods such as genetic algorithms dthe adp approach based on parametric and nonparametric approximations while producing reasonable solutions is not competitive overall this research illustrates that the base heuristic approach is a promising computational approach for mkps worthy of further investigation

sufficient conditions for the solubility of inverse eigenvalue problems abstract we define an inverse eigenvalue problem which contains as special cases the classical additive and multiplicative inverse eigenvalue problems using some results on the distance of eigenvalues from matrix diagonal elements and brouwer s fixed point theorem we give sufficient conditions for the solubility of the problem

inverse structural modification using constraints in a structural modification problem the mass and stiffness matrices are modified to obtain a desired spectrum in this paper this is done by imposing constraints on the structure the undamped natural vibrations of a constrained linear structure are calculated by solving a generalized eigenvalue problem derived from the equations of motion for the constrained system involving lagrangian multipliers the coefficients of the constraint matrix are taken as design variables and a set of equations defining the inverse structural modification problem is formulated this modification problem requires an iterative method for its solution an algorithm based on newton s method is employed each iteration step involves the calculation of a rectangular jacobian and the solving of an associated underdetermined system of linear equations the system can be solved by using the moore penrose inverse the method is demonstrated in some numerical examples

an analysis of stochastic shortest path problems we consider a stochastic version of the classical shortest path problem whereby for each node of a graph we must choose a probability distribution over the set of successor nodes so as to reach a certain destination node with minimum expected cost the costs of transition between successive nodes can be positive as well as negative we prove natural generalizations of the standard results for the deterministic shortest path problem and we extend the corresponding theory for undiscounted finite state markovian decision problems by removing the usual restriction that costs are either all nonnegative or all nonpositive

a quadratically convergent algorithm for inverse eigenvalue problems with multiple eigenvalues abstract in 2017 for inverse symmetric eigenvalue problems a new quadratically convergent algorithm has been derived from simple matrix equations although this algorithm has some nice features compared with the other quadratically convergent methods it is not applied to multiple eigenvalues in this paper we improve this algorithm with the aid of an optimization problem for the eigenvectors associated with multiple eigenvalues the proposed algorithm is adapted to an arbitrary set of given eigenvalues the main contribution is our convergence theorem formulated in a different manner from previous work for the existing quadratically convergent methods our theorem ensures the quadratic convergence in a neighborhood of the solutions that satisfy a mild condition

a newton iteration process for inverse eigenvalue problems we suppose an inverse eigenvalue problem which includes the classical additive and multiplicative inverse eigenvalue problems as special cases for the numerical solution of this problem we propose a newton iteration process and compare it with a known method finally we apply it to a numerical example

relaxing dynamic programming the idea of dynamic programming is general and very simple but the curse of dimensionality is often prohibitive and restricts the fields of application this paper introduces a method to reduce the complexity by relaxing the demand for optimality the distance from optimality is kept within prespecified bounds and the size of the bounds determines the computational complexity several computational examples are considered the first is optimal switching between linear systems with application to design of a dc dc voltage converter the second is optimal control of a linear system with piecewise linear cost with application to stock order control finally the method is applied to a partially observable markov decision problem pomdp

an efficient algorithm based on lanczos type of bcr to solve constrained quadratic inverse eigenvalue problems abstract this paper proposes the lanczos type of biconjugate residual bcr algorithm for solving the quadratic inverse eigenvalue problem l 2 x 2 l 1 x l 0 x 0 where l 2 l 1 and l 0 should be partially bisymmetric under a prescribed submatrix constraint an analysis reveals that the algorithm obtains the solutions of the constrained quadratic inverse eigenvalue problem in finitely many steps in the absence of round off errors finally numerical results are performed to confirm the analysis and to illustrate the efficiency of the proposed algorithm

generalized polynomial approximations in markovian decision processes abstract fitting the value function in a markovian decision process by a linear superposition of m basis functions reduces the problem dimensionality from the number of states down to m with good accuracy retained if the value function is a smooth function of its argument the state vector this paper provides for both the discounted and undiscounted cases three algorithms for computing the coefficients in the linear superposition linear programming policy iteration and least squares

an algorithm for symmetric generalized inverse eigenvalue problems abstract using qr like decomposition with column pivoting and least squares techniques we propose a new and efficient algorithm for solving symmetric generalized inverse eigenvalue problems and give its locally quadratic convergence analysis we also present some numerical experiments which illustrate the behaviour of our algorithm

relaxation analysis for the dynamic knapsack problem with stochastic item sizes we consider a version of the knapsack problem in which an item size is random and revealed only when the decision maker attempts to insert it after every successful insertion the decision maker ca

semi infinite relaxations for the dynamic knapsack problem with stochastic item sizes we consider a version of the knapsack problem in which an item size is random and revealed only when the decision maker attempts to insert it after every successful insertion the decision maker can choose the next item dynamically based on the remaining capacity and available items while an unsuccessful insertion terminates the process we propose a new semi infinite relaxation based on an affine value function approximation and show that an existing pseudo polynomial relaxation corresponds to a nonparametric value function approximation we compare both theoretically to other relaxations from the literature and also perform a computational study our main empirical conclusion is that our new relaxation provides tight bounds over a variety of different instances and becomes tighter as the number of items increases

a cost shaping linear program for average cost approximate dynamic programming with performance guarantees we introduce a new algorithm based on linear programming for optimization of average cost markov decision processes mdps the algorithm approximates the differential cost function of a perturbed mdp via a linear combination of basis functions we establish a bound on the performance of the resulting policy that scales gracefully with the number of states without imposing the strong lyapunov condition required by its counterpart in de farias and van roy de farias d p b van roy 2003 the linear programming approach to approximate dynamic programming oper res 51 6 850 865 we investigate implications of this result in the context of a queueing control problem

an infinite dimensional linear programming algorithm for deterministic semi markov decision processes on borel spaces we devise an algorithm for solving the infinite dimensional linear programs that arise from general deterministic semi markov decision processes on borel spaces the algorithm constructs a sequence of approximate primal dual solutions that converge to an optimal one the innovative idea is to approximate the dual solution with continuous piecewise linear ridge functions that naturally represent functions defined on a high dimensional domain as linear combinations of functions defined on only a single dimension this approximation gives rise to a primal dual pair of semi infinite programs for which we show strong duality in addition we prove various properties of the underlying ridge functions

stochastic shortest path games we consider dynamic two player zero sum games where the minimizing player seeks to drive an underlying finite state dynamic system to a special terminal state along a least expected cost path the maximizer seeks to interfere with the minimizer s progress so as to maximize the expected total cost we consider for the first time undiscounted finite state problems with compact action spaces and transition costs that are not strictly positive we admit that there are policies for the minimizer which permit the maximizer to prolong the game indefinitely under assumptions which generalize deterministic shortest path problems we establish i the existence of a real valued equilibrium cost vector achievable with stationary policies for the opposing players and ii the convergence of value iteration and policy iteration to the unique solution of bellman s equation

a backward error for the symmetric generalized inverse eigenvalue problem abstract in this paper we discuss the backward error for the symmetric generalized inverse eigenvalue problem which extends the result of sun 1999 10 the optimal backward error is defined for symmetric generalized inverse eigenvalue problem with respect to an approximate solution and the upper and lower bounds are derived for the optimal backward error the expressions may be useful for testing the stability of practical algorithms

least squares policy iteration we propose a new approach to reinforcement learning for control problems which combines value function approximation with linear architectures and approximate policy iteration this new approach is motivated by the least squares temporal difference learning algorithm lstd for prediction problems which is known for its efficient use of sample experiences compared to pure temporal difference algorithms heretofore lstd has not had a straightforward application to control problems mainly because lstd learns the state value function of a fixed policy which cannot be used for action selection and control without a model of the underlying process our new algorithm least squares policy iteration lspi learns the state action value function which allows for action selection without a model and for incremental policy improvement within a policy iteration framework lspi is a model free off policy method which can use efficiently and reuse in each iteration sample experiences collected in any manner by separating the sample collection method the choice of the linear approximation architecture and the solution method lspi allows for focused attention on the distinct elements that contribute to practical reinforcement learning lspi is tested on the simple task of balancing an inverted pendulum and the harder task of balancing and riding a bicycle to a target location in both cases lspi learns to control the pendulum or the bicycle by merely observing a relatively small number of trials where actions are selected randomly lspi is also compared against q learning both with and without experience replay using the same value function architecture while lspi achieves good performance fairly consistently on the difficult bicycle task q learning variants were rarely able to balance for more than a small fraction of the time needed to reach the target location

duality and existence of optimal policies in generalized joint replenishment we establish a duality theory for a broad class of deterministic inventory control problems on continuous spaces that includes the classical joint replenishment problem and inventory routing using this theory we establish the existence of an optimal policy which has been an open question we show how a primal dual pair of infinite dimensional linear programs encode both cyclic and noncyclic schedules and provide various results regarding cyclic schedules including an example showing that they need not be optimal

the linear programming approach to approximate dynamic programming the curse of dimensionality gives rise to prohibitive computational requirements that render infeasible the exact solution of large scale stochastic control problems we study an efficient method based on linear programming for approximating solutions to such problems the approach fits a linear combination of pre selected basis functions to the dynamic programming cost to go function we develop error bounds that offer performance guarantees and also guide the selection of both basis functions and state relevance weights that influence quality of the approximation experimental results in the domain of queueing network control provide empirical support for the methodology

approximate linear programming for average cost mdps we consider the linear programming approach to approximate dynamic programming with an average cost objective and a finite state space using a lagrangian form of the linear program lp the average cost error is shown to be a multiple of the best fit differential cost error this result is analogous to previous error bounds for a discounted cost objective second bounds are derived for average cost error and performance of the policy generated from the lp that involve the mixing time of the markov decision process mdp under this policy or the optimal policy these results improve on a previous performance bound involving mixing times

inverse eigenvalue problem in structural design a method is presented for the design of a structure with specified low order natural frequencies the method can also be used for design problems in which certain bands devoid of any frequency constraints are specified this can further be used to generate initial feasible designs for optimum design problems with frequency constraints

reductions of approximate linear programs for network revenue management the linear programming approach to approximate dynamic programming has received considerable attention in the recent network revenue management literature a major challenge of the approach lies in solving the resulting approximate linear programs alps which often have a huge number of constraints and or variables we show that the alps can be dramatically reduced in size for both affine and separable piecewise linear approximations to network revenue management problems under both independent and discrete choice models of demand our key result is the equivalence between each alp and a corresponding reduced program which is more compact in size and admits an intuitive probabilistic interpretation for the affine approximation to network revenue management under an independent demand model we recover an equivalence result known in the literature but provide an alternative proof our other equivalence results are new we test the numerical performance of solving the reduced programs directly using o

a vehicle routing problem with stochastic demand we consider a natural probabilistic variation of the classical vehicle routing problem vrp in which demands are stochastic given only a probabilistic description of the demand we need to design routes for the vrp motivated by applications in strategic planning and distribution systems rather than resolving the problem when the demand becomes known we propose to construct an a priori sequence among all customers of minimal expected total length we analyze the problem using a variety of theoretical approaches we find closed form expressions and algorithms to compute the expected length of an a priori sequence under general probabilistic assumptions based on these expressions we find upper and lower bounds for the probabilistic vrp and the vrp re optimization strategy in which we find the optimal route at every instance we propose heuristics and analyze their worst case performance as well as their average behavior using techniques from probabilistic analysis our results suggest that our approac

on thec1stability conjecture for flows we give a proof of thec1stability conjecture of palis and smale for flows which has reduced to proving thatc1structural stability implies axiom a the proof is based on the fundamental work of liao and mane and on the recent powerfulc1connecting lemma of hayashi

maximizing a class of submodular utility functions given a finite ground set n and a value vector a in mathbb r n we consider optimization problems involving maximization of a submodular set utility function of the form h s f left sum i in s a i right s subseteq n where f is a strictly concave increasing differentiable function this utility function appears frequently in combinatorial optimization problems when modeling risk aversion and decreasing marginal preferences for instance in risk averse capital budgeting under uncertainty competitive facility location and combinatorial auctions these problems can be formulated as linear mixed 0 1 programs however the standard formulation of these problems using submodular inequalities is ineffective for their solution except for very small instances in this paper we perform a polyhedral analysis of a relevant mixed integer set and by exploiting the structure of the utility function h strengthen the standard submodular formulation significantly we show the lifting problem of the submodular inequalities to be a submodular maximization problem with a special structure solvable by a greedy algorithm which leads to an easily computable strengthening by subadditive lifting of the inequalities computational experiments on expected utility maximization in capital budgeting show the effectiveness of the new formulation

a quadratically convergent algorithm based on matrix equations for inverse eigenvalue problems abstract we propose a quadratically convergent algorithm for inverse symmetric eigenvalue problems based on matrix equations the basic idea is seen in a recent study by ogita and aishima while they derive an efficient iterative refinement algorithm for symmetric eigenvalue problems using special matrix equations in other words this study is interpreted as a unified view on quadratically convergent algorithms for eigenvalue problems and inverse eigenvalue problems based on matrix equations to the best of our knowledge such a unified development of algorithms is provided for the first time since the proposed algorithm for the inverse eigenvalue problems can be regarded as the newton s method for the matrix equations the quadratic convergence is naturally proved our algorithm is interpreted as an improved version of the cayley transform method for the inverse eigenvalue problems although the cayley transform method is one of the effective iterative methods the cayley transform takes o n 3 arithmetic operations to produce an orthogonal matrix using a skew symmetric matrix in each iteration our algorithm can refine orthogonality without the cayley transform which reduces the operations in each iteration it is worth noting that our approach overcomes the limitation of the cayley transform method to the inverse standard eigenvalue problems resulting in an extension to inverse generalized eigenvalue problems

a polynomial membership function approach for stability analysis of fuzzy systems for the stability analysis of a polynomial fuzzy system a new polynomial membership function approach is proposed to reduce the conservatism based on a state feedback closed loop system a polynomials fitting method is utilized and an improved membership functions transformation technique are proposed to approximate the membership functions of fuzzy system then the membership function dependent polynomials form stability conditions are derived the obtained polynomial membership functions and approximation errors will be involved in the stability analysis process based on the sum of squares optimization technique the polynomials conditions can be directly solved finally by several numerical and practical examples the conservatism reduction effects are shown by comparisons with existing methods