Rejoinder - The Languages of Stochastic Optimization

The theme of my feature article is the crossfertilization of ideas from artificial intelligence and operations research, although it is important to emphasize that some of the key ideas of approximate dynamic programming also draw heavily on developments whose origins should be attributed to control theory. I am very grateful for the insightful remarks of John N. Tsitsiklis, well known for his many contributions to approximate dynamic programming and his coauthorship of a seminal book (Bertsekas and Tsitsiklis 1996), and Andrzej Ruszczynski, one of the very top names in the field of stochastic programming who, in addition to numerous research papers, coedited the volume on stochastic programming in the prestigious Handbook on Operations Research and Management Science series (Ruszczynski and Shapiro 2003). I also benefited from extensive comments by Andy Barto who, with Richard Sutton, coauthored the book Reinforcement Learning, which is the bible of this hugely popular field (Sutton and Barto 1998). Both sets of comments add a number of useful insights. I am going to focus my final remarks less on the specifics of their remarks and more on the differences in perspectives they bring from their home communities. I also need to note that there is no such thing as the “operations research perspective”— rather, there are at least four major subcommunities within operations research that deal directly with stochastic optimization: Markov decision processes (MDPs) (as represented by Puterman 1994), stochastic programming (Kall and Wallace 1994, Higle and Sen 1996, and Birge and Louveaux 1997 are major references), decision theory (primarily decision trees, as in Skinner 1999), and simulation-optimization (see, for example, Swisher et al. 2000, Fu 2002). Tsitsiklis (2010) notes the strong similarities between operations research and the field of systems and control theory. In drawing on this conclusion, he is focusing on the central paradigm of optimizing a known objective function with known system dynamics. Artificial intelligence (AI), by contrast, is often addressing the problem of learning how people behave (or more fundamentally, how the brain solves problems), often in the context of applications where we may not understand why someone is making a particular decision (hence, we may not know the reward function). Also, these applications often take place in an online setting where we are observing events from an external, physical system. As a result, we may be able to observe the next state we land in, but we do not necessarily know the physics of why we landed there. The first difference someone will notice between these communities appears purely cosmetic. The AI and MDP communities use state s and action a; elsewhere, operations research will typically use x for a decision, expecting it to be a high-dimensional vector. In control theory, the state is x and the decision (control) is u. Behind these cosmetic differences are problem classes with associated empirical properties and computational expectations. A 10-dimensional control problem can be much harder than an OR problem with 10,000 dimensions. A second and significant difference between the communities is how they represent the dynamics of the system. In the MDP community, it is absolutely routine to start by knowing the transition matrix p s′ s x , which gives the probability of landing in (typically discrete) state s′, given that we are in state s and take action x. However, there are many problems where this matrix is either simply unknown or computationally intractable (for example, if the number of states is too large). Computing the transition matrix requires first knowing the transition function, which might be written as (using my notation)