A Hierarchical Adaptive Approach to the Optimal Design of Experiments.

A Hierarchical Adaptive Approach to the Optimal Design of Experiments Woojae Kim 1 (kim.1124@osu.edu), Mark Pitt 1 (pitt.2@osu.edu), Zhong-Lin Lu 1 (lu.535@osu.edu), Mark Steyvers 2 (mark.steyvers@uci.edu), Hairong Gu 1 (gu.124@osu.edu), & Jay Myung 1 (myung.1@osu.edu) Department of Psychology, Ohio State University, Columbus, OH 43210 USA Department of Cognitive Sciences, University of California, Irvine, CA 92697 USA by adaptively taking into account what has been learned about the participant’s performance from past trials. Abstract Contrast Sensitivity Experimentation is at the core of research in cognitive science, yet observations can be expensive and time- consuming to acquire. A major interest of researchers is designing experiments that lead to maximal accumulation of information about the phenomenon under study with the fewest possible number of observations. In addressing this challenge, statisticians have developed adaptive design optimization methods. This paper introduces a hierarchical Bayes extension of adaptive design optimization that provides a judicious way to exploit two complementary schemes of inference (with past and future data) to achieve even greater accuracy and efficiency in information gain. We demonstrate the method in a simulation experiment in the field of visual perception. Invisible Visible Spatial Frequency Keywords: optimal experimental design, hierarchical Bayes, mutual information, visual spatial processing. Figure 1: Examples of stimuli (left) and a typical contrast sensitivity function (right). Sinewave gratings with varying contrast and spatial frequency are used to measure a person’s CSF. Introduction Experimentation advances cognitive science by providing quantified evidence for evaluating and testing theories. From the standpoint of information theory (Cover & Thomas, 1991), one should design an experiment that across trials seeks to gain as much information as possible from the cognitive process under study. For example, experiments often require choosing levels of an independent variable (e.g., stimuli of different sizes or intensities). These choices impact the informativeness (or quality) of the resulting data, which in turn impacts what can be concluded about the issue of interest. As a concrete example, consider an experiment in visual psychophysics in which one is interested in estimating a viewer’s ability to see fine detail. When the sensitivity is measured with stimuli that vary not only in their contrast but also in their spatial frequency, the measurements form a contrast sensitivity function (CSF; Figure 1). A CSF characterizes a person’s vision more accurately than traditional visual acuity measurements (i.e., using an eye chart) and is often useful for detecting visual pathologies ( Comerford, 1983 ). However, because the standard methodology (e.g., staircase procedure) can require many hundreds of trials for accurate estimation of the CSF curve, it is a prime candidate for improving information gain, and Lesmes, Lu, Baek and Albright (2010) developed such a method. In each trial of the experiment, the to-be-presented stimulus is chosen such that it maximizes information gain Their procedure was one implementation of algorithm technology being developed in the burgeoning field of design optimization (Atkitson & Donev, 1992; Amzal, Bois, Parent, & Robert, 2006). Called adaptive design optimization (ADO; e.g., DiMattina & Zhang, 2008; Cavagnaro, Tang, Myung & Pitt, 2009), this method capitalizes on the sequential nature of experimentation, making each new measurement using the information learned from previous measurements of a subject so as to achieve maximal information about the cognitive process under study. As currently used, ADO is tuned to optimizing a measurement process at the individual participant level, without taking advantage of information available from data collected from other individuals or testing sessions. To illustrate the situation using the example of CSF estimation, consider a space of CSFs in Figure 2, a point in which represents an individual’s measured CSF (e.g., the ones shown with arrows). Suppose that one has already collected CSFs from a group of participants (dots in the ellipse) and data are about to be collected from one more (triangle). Knowledge of what this person’s CSF function might look like can be informed by the group data, and thereby expedite and improve data collection. Without the benefit of the group-level information, data collection would be more