Capacity limits and how the visual system copes with them

A visual system cannot process everything with full fidelity, nor, in a given moment, perform all possible visual tasks. Rather, it must lose some information, and prioritize some tasks over others. The human visual system has developed a number of strategies for dealing with its limited capacity. This paper reviews recent evidence for one strategy: encoding the visual input in terms of a rich set of local image statistics, where the local regions grow — and the representation becomes less precise — with distance from fixation. The explanatory power of this proposed encoding scheme has implications for another proposed strategy for dealing with limited capacity: that of selective attention, which gates visual processing so that the visual system momentarily processes some objects, features, or locations at the expense of others. A lossy peripheral encoding offers an alternative explanation for a number of phenomena used to study selective attention. Based on lessons learned from studying peripheral vision, this paper proposes a different characterization of capacity limits as limits on decision complexity. A general-purpose decision process may deal with such limits by “cutting corners” when the task becomes too complicated. Human vision is full of puzzles. Observers can grasp the essence of a scene in less than 100 ms, reporting with a fair degree of reliability whether it is a beach or a street, whether it contains any animals, and what materials are present [1, 2]. Yet when probed for details, they are at a loss. Change the scene while masking the motion transients, and the observer may have great difficulty determining what has changed, even when the change is quite visible once it has been spotted (“change-blindness”, [3, 4]). Human vision is better than the best computer vision systems ever created, yet it is also easily fooled by visual illusions. People can look at a line drawing of a 3D object, and effortlessly understand its shape, yet have difficulty noticing the impossibility of an Escher never-ending staircase. We have difficulty finding our keys, even when they prove quite visible once found and fixated, and look nothing like other items on our desk. How does one explain this combination of marvelous successes and quirky failures? It perhaps seems unsurprising that these diverse phenomena at present have no unifying explanation. What do they have in common? Certainly, scene perception, object recognition, and 3-D shape estimation require different mechanisms at some stage of visual processing. Nonetheless, might there exist a coherent explanation in terms of a critical stage of processing that determines performance for a wide variety of tasks, or at least a guiding principle for what tasks are easy and difficult? Attempts to provide a unifying account have explained the failures in terms of the visual system having limited capacity (see [5] for a review). Our senses gather copious amounts of data, seemingly far more than our minds can fully process at once. At any given instant, we are consciously aware of only a small fraction of the incoming sensory input. We seem to have a limited capacity for awareness, for memory, and for the number of tasks we can simultaneously carry out, leading to poor performance at tasks that stress the capacity limits of the system. A classic example of the limited capacity logic concerns visual search. Suppose a researcher runs an experiment in which observers must find a target item among a number of other “distractor” items. As in many such experiments, the experimenter picks a target and distractors such that individual items seem easy to distinguish. Nonetheless, the researcher finds that search is inefficient, i.e., that it becomes significantly slower as one adds more distractors. Why is search difficult? One can easily discriminate the target from the distractors when looking directly at them. The poor search performance implies that vision is not the same everywhere, or, as Julian Hochberg put it, “vision is not everywhere dense” [6]. If vision were the same throughout the visual field, search would be easy. By a popular account, the main reason vision is not the same everywhere has to do with attention, in particular selective attention. In this account, attention is a limited resource, and vision is better where the observer attends than where they do not. The visual system deals with limited capacity by serially shifting attention. Some tasks require selective attention, and as a result are subject to the performance limits inherent in having to wait for this limited resource. In the case of difficult search tasks, for instance, the targetdistractor discrimination is presumed to require attention, making search significantly slower with increasing number of display items. On the other hand, preattentive tasks do not require attention; they can be performed quickly and in parallel, leading to easy search. Selective attention is typically described as a mechanism that gates access to further visual processing [7, 8, 9] rather than engaging in processing itself. Once the visual system selects a portion of the visual input, perception happens. Throughout this paper, when I refer to selective attention, I mean a gating mechanism. Traditionally, researchers have taken visual search phenomena as evidence that selective attention operates early in the visual processing pipeline, and that correct binding of basic features into an object requires selective attention [10]. Though this account has had a certain amount of predictive power when it comes to visual search, it has been problematic overall [11, 12, 13, 14]. The need for selective attention to bind basic features seems to conflict with: the relative ease of searching for a cube among differently lit cubes [15, 16, 17]; with easy extraction of the gist of a scene [18, 19, 2, 20, 21, 22, 23] and of ensemble properties of sets [24, 25, 26]; and with what tasks require attention in a dual-task paradigm [27]. My lab has argued instead that a main way in which the visual system deals with limited capacity is through encoding its inputs in a way that favors foveal vision over peripheral. Peripheral vision is, as a rule, worse than foveal vision, and often much worse. Peripheral vision must condense a mass of information into a succinct representation that nonetheless carries the information needed for vision at a glance. Only a finite number of nerve fibers can emerge from the eye, and rather than providing uniformly mediocre vision, the eye trades off sparse sampling in the periphery for sharp, high resolution foveal vision. This economical design continues into the cortex: more cortical resources are devoted to processing central vision at the expense of the periphery. We have proposed that the visual system deals with limited capacity in part by representing its input in terms of a rich set of local image statistics, where the local regions grow — and the representation becomes less precise — with distance from fixation [28]. Such a summary-statistic representation would render vision locally ambiguous in terms of the phase and location of features. Thus, this scheme trades off computation of sophisticated image features at the expense of spatial localization of those features. One of the main implications of this theory for vision science has been the need to re-examine understanding of visual attention. Most experiments investigating selective attention have had a peripheral vision confound. A number of phenomena previously attributed to attention may instead arise in large part from peripheral encoding. This paper begins by reviewing both phenomena in peripheral vision and our model of peripheral encoding. It reviews what we have learned about perception, as well as the implications for theories of attention, particularly selective attention. Our understanding of peripheral vision constrains possible additional mechanisms for dealing with limited capacity. In particular, I propose that the brain may face limits on decision complexity, and deal with those limits by performing a simpler version of any toocomplex task, leading to poorer performance at the nominal task. A lossy encoding in peripheral vision Peripheral vision is susceptible to clutter, as evidenced by the phenomena of visual crowding. Classic crowding refers to greater difficulty identifying a peripheral target when flanked by neighboring stimuli than when it appears in isolation. Crowded stimuli may appear jumbled and uncertain, lacking crucial aspects of form, almost as if they have a textural or statistical nature [29]. Crowding has often been studied with a target identification task, and with a target object flanked by other objects, but it almost certainly affects perception more generally. Crowding points to significant qualitative differences between foveal and peripheral vision. These differences are far greater than the modest differences between foveal and peripheral acuity, and are likely task-relevant for a wide variety of tasks [30]. The phenomena of crowding have been described in detail in a number of recent review papers [31, 32, 33, 34]. My lab has argued that one must control for or otherwise account for the strengths and limitations of peripheral vision before considering explanations based upon visual attention [30, 14, 35]. Otherwise, one risks fundamental misunderstandings about both perception and attention. Whether the paradigm is visual search, change detection, dual-task, scene perception, or inattentional blindness – all tasks whose results have been interpreted in terms of the mechanisms of attention – the often-cluttered stimuli lie at least in part outside of the fovea, and are potentially subject to crowding. A number of researchers have suggested that crowding results from “forced texture perception,” in which information is pooled over sizeable portions of the visual field [29, 36, 31, 32]. Based on these intuit