Controlling State-Space Explosion in Chunk Learning

Controlling State-Space Explosion in Chunk Learning Vladislav D. Veksler , Kevin A. Gluck , Christopher W. Myers , Jack Harris , Thomas Mielke Air Force Research Laboratory, Wright-Patterson AFB, USA Abstract Varying combinations of perceptual cues may be relevant for learning and action-selection. However, storing all possible cue combinations in memory is computationally implausible in sufficiently complex environments due to a state-space ex- plosion. Some psychological models suggest that cue combi- nations, i.e. chunks, should be generated at a conservative rate (EPAM/CHREST; e.g. Feigenbaum & Simon, 1984). Other models suggest that chunk retrieval is based on statistical reg- ularities in the environment (i.e. recency and frequency; An- derson & Schooler, 1991). We present a computational model of chunk generation based on these two principles, and demon- strate how combining these principles alleviates state-space ex- plosion, producing great savings in memory while maintaining a high level of performance. Keywords: chunking, unitization, configural-cue, state-space explosion, combinatoric explosion, learning, rational analysis Introduction A brown apple is neither just brown, nor just an apple, nor just a brown apple. It is all of these things simultaneously, and any of these representations may be important for both learning and action-selection. Concurrent representation of all cue combinations allows for learning of both generic rules (e.g. apples taste good) and exceptions to those rules (e.g. brown apples are spoiled). In the psychological literature a set of perceptual cues is known as a chunk (e.g. Feigenbaum & Simon, 1984) or a configural-cue (Wagner & Rescorla, 1972). Having a single memory chunk for representing a given combination of cues allows for faster recognition of current state and faster action selection (e.g. Goldstone, 2000). Chunk representation also aids in the ability to store greater amounts of information in working memory (Ericsson, Delaney, Weaver, & Mahadevan, 2004). A commonly cited example of this phenomenon in- volves improved recognition of displayed chess positions by chess experts compared to novices (Chase & Simon, 1973; Gobet, 1998). The theory is that experts will have been ex- posed to more chess situations, and will have learned many chunks for representing complex states on the chess board. Thus a chess expert will need fewer chunks to represent a chess position than a novice, who may have to memorize the position as a series of figures and their locations. This theory is further supported by the fact that experts are no better than novices in recall of chess boards where pieces are placed ran- domly (rather than positions encountered during actual game- play). Each set of perceptual cues may potentially be recog- nized as multiple concurrently active chunks. Cues large, square, and white may activate seven different chunks: {large}, {square}, {white}, {large, square}, {large, white}, {square, white}, and {large, square, white}. Recognition of a set of perceptual cues as multiple concurrently active chunks can account for a wide range of behavioral phenomena, in- cluding classical conditioning (Pearce, 1994), base-rate ne- glect (Gluck & Bower, 1988a, 1988b), and categorization (e.g. Gluck & Bower, 1988b; Veksler, Gray, & Schoelles, The problem with a full chunk representation, where each set of cues activates all possible combinations of those cues in memory, is that in complex environments too many chunks may be required. A mere ten binary perceptual inputs (e.g. black vs white, large vs small) may require over 59 thousand chunks to be present in memory 1 . If each perceptual input allowed for five possible values (e.g. black, dark-grey, grey, light-grey, white), ten such input dimensions could result in almost ten million chunks. Twenty such perceptual dimen- sions would result in 95 trillion chunks. One hundred inputs with ten values per input would result in more chunks than there are atoms in the observable universe. Given these num- bers, it is safe to say that the brain does not create a memory element for each combination of input signals, and a com- putational system based on a full chunk representation is not feasible in complex environments. The exponential growth of memory based on combina- tions of perceptual cues is called the state-space explosion problem. This problem becomes even more severe when a chunk-based memory system is integrated with State-Action and/or State-State connections (e.g. Gluck & Bower, 1988b; Pearce, 1994; Veksler et al., 2007), where the total amount of required memory may be an exponential function of the number of chunks, creating double-exponential mem- ory growth. There is much psychological evidence that humans do not hold memory representations for all potential combinations of situational features. Rather, cues get merged over time to make more and more complex object representations. There is evidence of this phenomenon in the context of percep- tual unitization within hour-long psychological experiments (e.g. Goldstone, 2000), as well as in life-long acquisition of subject-matter expertise (e.g. Feigenbaum & Simon, 1984). In this paper we present a model of gradual chunk learn- ing based on widely supported principles of human mem- ory, and examine how this helps to alleviate the state-space Given n cues (e.g. large, square, white), we can create a chunk for every combination of cue presence and absence ({large}, {square}, {white}, {large, square}, {large, white}, {square, white}, and {large, square, white}). If we represent cue presence as a 1 and cue absence as a 0, we can represent each chunk as a binary number, and the total number of possible chunks is the total number of possi- ble binary numbers (minus the blank chunk), which is 2 n − 1. When each cue dimension can have two potential values, the total number of possible chunks is 3 n − 1. With k − 1 possible values on n feature dimensions, we can have at most k n − 1 possible chunks to represent all potential cue combinations.