Further thoughts on the paper by Reverdy , Srivastava , and Leonard On Human Decision-Making Heuristics and Algorithm Design in Engineering

Automation is playing an increasingly significant role in growing areas from healthcare and environmental monitoring to alternative energy and manufacturing. This requires methodologies for automated agents, such as supervisory or in situ robotic systems, to reliably and efficiently interact in real time with complex, uncertain environments. In turn, these methodologies rely critically on decision-making algorithms with provable performance that successfully address the many tensions that arise from multiple performance objectives, including, for example, the tradeoffs between speed and accuracy and between efficiency and robustness to failures and uncertainties. Engineering approaches to algorithm design have been developed, and the literature continues to grow with advances in both theory and technology. However, online learning and optimal decision and control theories tend to run up against limitations such as the ‘‘curse of dimensionality,’’ which can confound the design, analysis, and verification of algorithms in even moderately complex environments. Humans are routinely faced with decision-making challenges with similar tensions. In many circumstances, humans can perform well, particularly if they have skill or training. Neuroscience research sheds light on the mechanisms of cognitive control, i.e., the ability to guide attention, thought, and action with respect to goals or intentions [1], which in turn can be used to explain and justify heuristics employed by humans. Because these heuristics have the advantage of both computational tractability and biological plausibility, there is great promise in leveraging them in the design of algorithms. In our paper, we derived provable algorithms for decision making under uncertainty drawing from research on heuristics used by humans in the tradeoff between exploration and exploitation. Exploitation refers to making choices among well-known alternatives, whereas exploration refers to making choices among uncertain but possibly better alternatives. Humans make decisions every day that require exploring options in advance of exploiting what is known. Many factors come into play such as prior knowledge and the number of future choices. The multiarmed bandit problem provides a generalized framework for studying the explore versus exploit tension. Each of the N arms of the bandit refers to one of N alternatives, and each arm has associated to it a reward that the decision maker receives if the arm is chosen. The decision-making task is to choose a finite sequence of arms to maximize accumulated reward. In our paper, the reward for each arm is a Gaussian variable with unknown mean and known variance. The multiarmed bandit framework has been used in neuroscience research to examine human explore–exploit heuristics. For example, in [2], Wilson et al. showed in the N 1⁄4 2 case that humans choose the alternative with the largest weighted sum of estimated reward and information gain, where the weight depends on factors such as the future number of choices. In [3], Acuña and Schrater showed that humans leverage their ability to learn the correlation structure among the different alternatives. An approach known as ‘‘optimism in the face of uncertainty’’ is used to derive optimal strategies in the multiarmed bandit literature in engineering [4]. The idea is to choose among alternatives based on a credible estimate of the highest possible reward for each alternative. This produces heuristics much like the heuristics used by humans in [2]. The stochastic upper credible limit (UCL) algorithm developed in our paper connects the studied human heuristics with a decision maker that uses Bayesian inference together with the ‘‘optimism in the face of uncertainty’’ strategy, following [5]. Other features of our algorithm reflective of human behavior include decision-making noise, the role of finite time horizon, and familiarity with the environment and its correlation structure. We proved conditions for high performance, and we showed how the algorithm efficiently captures different categories of observed human performance. Digital Object Identifier: 10.1109/JPROC.2014. 2308607