Global Model Analysis of Cognitive Variability

Residual fluctuations produced in typical experimental methodologies are examined as correlated noises. The effective range of the correlations was assessed by determining whether the decay over look-back time is better described as a power law or exponential. Both of these decay laws contain free parameters and it is argued that it is not possible to distinguish their models on the basis of simple measures of goodness-of-fit. Global analyses that evaluate models on the basis of how well they generalize are conducted. The models are examined in terms of three constructs that all bear on generalization: cross-validity, flexibility, and representativeness. Quantitative assessment of a large ensemble of data suggests that the correlations decay over time as a power law. The conclusion is that human residual fluctuation is a correlated fractal.

Related papers:

title: Nonlinear forecasting as a way of distinguishing chaos from measurement error in time series
abstract: An approach is presented for making short-term predictions about the trajectories of chaotic dynamical systems. The method is applied to data on measles, chickenpox, and marine phytoplankton populations, to show how apparent noise associated with deterministic chaos can be distinguished from sampling error and other sources of externally induced environmental noise.

title: Physiological time-series analysis using approximate entropy and sample entropy.
abstract: Entropy, as it relates to dynamical systems, is the rate of information production. Methods for estimation of the entropy of a system represented by a time series are not, however, well suited to analysis of the short and noisy data sets encountered in cardiovascular and other biological studies. Pincus introduced approximate entropy (ApEn), a set of measures of system complexity closely related to entropy, which is easily applied to clinical cardiovascular and other time series. ApEn statistics, however, lead to inconsistent results. We have developed a new and related complexity measure, sample entropy (SampEn), and have compared ApEn and SampEn by using them to analyze sets of random numbers with known probabilistic character. We have also evaluated cross-ApEn and cross-SampEn, which use cardiovascular data sets to measure the similarity of two distinct time series. SampEn agreed with theory much more closely than ApEn over a broad range of conditions. The improved accuracy of SampEn statistics should make them useful in the study of experimental clinical cardiovascular and other biological time series.

title: Fluctuations in the Time Required for Elementary Decisions
abstract: The nature of reaction time variability is analyzed in a suite of four experiments involving tasks, methodologies, and types of perceptual judgment commonly encountered in cognitive psychology In every case, a substantial fraction of the trial-by-trial variability in reaction time latency is shown to be well described by a particular type of fluctuation known as IA noise These results suggest that the time it takes to make and register a speeded decision reflects a kind of dynamic complexity that is seen in natural systems that self-organize at the boundary between order and chaos.

title: The emergent coordination of cognitive function.
abstract: 1/f scaling has been observed throughout human physiology and behavior, but its origins and meaning remain a matter of debate. Some argue that it is a byproduct of ongoing processes in the brain or body and therefore of limited relevance to psychological theory. Others argue that 1/f scaling reflects a fundamental aspect of all physiological and cognitive functions, namely, that they emerge in the balance of independent versus interdependent component activities. In 4 experiments, series of key-press responses were used to test between these 2 alternative explanations. The critical design feature was to take 2 measures of each key-press response: reaction time and key-contact duration. These measures resulted in 2 parallel series of intrinsic fluctuations for each series of key-press responses. Intrinsic fluctuations exhibited 1/f scaling in both reaction times and key-contact durations, yet the 2 measures were uncorrelated with each other and separately perturbable. These and other findings indicate that 1/f scaling is too pervasive to be idiosyncratic and of limited relevance. It is instead argued that 1/f scaling reflects the coordinative, metastable basis of cognitive function.

title: 1/f Fluctuation of Heartbeat Period
abstract: It is found that a heartbeat period fluctuation usually has a power spectral density which is inversely proportional to frequency, to which a spike is added at a breathing frequency.

title: 1/f model for long-time memory of the ocean surface temperature.
abstract: The 1/f spectrum of the ocean surface temperature in the Atlantic and Pacific midlatitudes is explained by a simple vertical diffusion model with a shallow mixed layer on top of a deep ocean. The model is forced at the air-sea interface with the total surface heat flux from a 1000 year climate simulation. The analysis reveals the role of ocean advection and substantiates estimates of internal thermal diffusivities.

title: Priming of pop-out: I. Role of features
abstract: We examined a visual search task, in which observers responded to the high-acuity aspect of a popout target (shape of an odd-colored diamond or vernier offset of an odd spatial-frequency patch). Repetition of the attention-driving feature (color or spatial frequency) in this task primes the popout; repetition of the high-acuity aspect (shape, vernier offset) does not. Priming of pop-out is due to a decaying memory trace of the attention-focusing feature laid down with each trial. The trace exerts a diminishing effect over the following five to eight trials (≈30 sec), and its influence over this time is cumulative. Observers cannot willfully overcome the priming, which suggests that it is passive and autonomous. Both target facilitation and distractor inhibition are evident; the former has a greater effect. The phenomenon shows complete binocular transfer.

title: Dynamic Models of Simple Judgments: I. Properties of a Self-Regulating Accumulator Module
abstract: This is the first of two papers comparing connectionist and traditional stochastic latency mechanisms with respect to their ability to account for simple judgments. In this paper, we show how the need to account for additional features of judgment has led to the formulation of progressively more sophisticated models. One of these, a self-regulating, generalized accumulator process, is treated in detail, and its simulated performance across a sample of tasks is described. Since an adaptive decision module of this kind possesses all the ingredients of intelligent behavior, it is eminently suited as a basic computing element in more complex networks.

title: Scalar Timing in Memory
abstract: A recent report of ours’ proposed an information-processing account of temporal generalization. The account posited a clock process, which was the basic time measurement device, and working and reference memory for storing the output of the clock either temporarily or relatively permanently. Records of time intervals in working and reference memory were then compared using a binary decision process, which dictated responding or not responding. The analysis concentrated on a relativistic Weber’s law property of the data from temporal generalization, and the constraints this property imposed on sources of variance in the information-processing stages. Our purpose here is to summarize that work and generalize the model in two ways: First we consider several sources of variance operating simultaneously. The original analysis demonstrated that if only one source of variance is present, it must be a scalar source, that is, it must result in a variable memory for which variance increases with the square of the mean.’ In the generalized account proposed here, we will develop the conclusion that scalar sources dominate in some time ranges, while other sources may dominate in others. These ideas are then applied to two additional timing tasks with different characteristics.

title: Response delays and the timing of discrete motor responses
abstract: A model for the timing of repetitive discrete motor responses is proposed, and a prediction of negative dependency between successive interresponse intervals is confirmed by data from a Morse key tapping task. A method that makes use of the first-order serial correlation between interresponse intervals is used to distinguish between variance due to a timekeeping process and variance in motor response delays subsequent to the timekeeper. These two quantities are examined as a function of mean interresponse interval.

title: 1/f noise: a pedagogical review
abstract: This is a pedagogical review of the ubiquitous 1/f^\alpha noises. The sections include the representation of 1/f^\alpha noise as a superposition of many relaxation processes; a discussion of the infinitely large fluctuations in the low frequency limit; 1/f^\alpha noise from diffusion processes; the measured statistical properties of 1/f^\alpha noises; 1/f^\alpha noises in self organized criticality; scaling laws in earthquake physics; 1/f noise in deterministic dynamical systems; an example of noise in a biophysical system; the numerical simulation of 1/f^\alpha noise; 1/f noise literature (including online resources).

title: Assessing the distinguishability of models and the informativeness of data
abstract: A difficulty in the development and testing of psychological models is that they are typically evaluated solely on their ability to fit experimental data, with little consideration given to their ability to fit other possible data patterns. By examining how well model A fits data generated by model B, and vice versa (a technique that we call landscaping), much safer inferences can be made about the meaning of a model's fit to data. We demonstrate the landscaping technique using four models of retention and 77 historical data sets, and show how the method can be used to: (1) evaluate the distinguishability of models, (2) evaluate the informativeness of data in distinguishing between models, and (3) suggest new ways to distinguish between models. The generality of the method is demonstrated in two other research areas (information integration and categorization), and its relationship to the important notion of model complexity is discussed.

title: Provenance of correlations in psychological data
abstract: Two distinct families of statistical processes are considered in the production of psychophysical time series data (Gilden, 1997, 2001; Gilden, Thornton, & Mallon, 1995). We inquire whether the spectral signatures of the underlying dynamics are better described in terms of short-range autoregressive movingaverage (ARMA) processes or long-range fractal processes. A thorough presentation of both families is given so as to clarify the scope and generalizability of the models as descriptions of choice reaction time data. Analyses of data are supplemented by the construction of a spectral likelihood classifier that discriminates between the two families of processes. The classifier has sufficient sensitivity to ensure that fractals are correctly identified and that ARMA processes will rarely be misconstrued as belonging to the fractal family. Spectral likelihood classification illustrates an extremely general framework for testing competing spectral hypotheses and is offered for use in measuring the specific character of fluctuations in designed experiments.

title: Universal 1/f Noise from Dissipative Self-Organized Criticality Models
abstract: We introduce a model able to reproduce the main features of 1/f noise: hyper-universality (the power-law exponents are independent on the dimension of the system; we show here results in d=1,2) and apparent lack of a low-frequency cutoff in the power spectrum. Essential ingredients of this model are an activation-deactivation process and dissipation.

title: ON THE UBIQUITY OF 1/f NOISE
abstract: A generic mechanism for the ubiquitous phenomenon of 1/f noise is reviewed. This mechanism arises in random processes expressible as a product of several random variables. Under mild conditions this product form leads to the log-normal distribution which we show straightforwardly generates 1/f noise. Thus, 1/f noise is tied directly to a probability limit distribution. A second mechanism involving scaling is introduced to provide a natural crossover from log-normal to inverse power-law behavior and generates 1/fα noise instead of pure 1/f noise. Examples of these distributions and the transitions between them are drawn from such diverse areas as economics, scientific productivity, bronchial structure and cardiac activity.

title: 28 The Long and Short of Timing in Response Sequences
abstract: This chapter is concerned with psychological processes underlying the timing of sequences of discrete motor responses. The continuation paradigm for the study of repetitive responding is described and a two-process model of the variation observed in self-paced interresponse intervals is reviewed. The model ascribes departures from periodic responding to the imprecision of a hypothetical timekeeper and to temporal “noise” in the execution of motor responses triggered by the timekeeper. Studies that throw light on the component processes of the model are discussed. The relevance of the model to the coordination of different phases of movement and to rhythm is shown.

title: Cross-Validation Methods.
abstract: This paper gives a review of cross-validation methods. The original applications in multiple linear regression are considered first. It is shown how predictive accuracy depends on sample size and the number of predictor variables. Both two-sample and single-sample cross-validation indices are investigated. The application of cross-validation methods to the analysis of moment structures is then justified. An equivalence of a single-sample cross-validation index and the Akaike information criterion is pointed out. It is seen that the optimal number of parameters suggested by both single-sample and two-sample cross-validation indices will depend on sample size.

title: I. Problems and Designs of Cross-Validation 1
abstract: THE term &dquo;cross-validation&dquo; is often loosely applied to any one of several distinct, though closely related, experimental designs. Before we get lost in a swamp of semantic confusion, it may be well to identify each of these, to point out their similarities and differences and to make clear the objectives which it serves. What name we give to each is secondary, although for convenience I shall attach a different name to each.

title: Cross‐Validatory Choice and Assessment of Statistical Predictions
abstract: A generalized form of the cross-validation criterion is applied to the choice and assessment of prediction using the data-analytic concept of a prescription. The examples used to illustrate the application are drawn from the problem areas of univariate estimation, linear regression and analysis of variance.

title: The Importance of Complexity in Model Selection.
abstract: Model selection should be based not solely on goodness-of-fit, but must also consider model complexity. While the goal of mathematical modeling in cognitive psychology is to select one model from a set of competing models that best captures the underlying mental process, choosing the model that best fits a particular set of data will not achieve this goal. This is because a highly complex model can provide a good fit without necessarily bearing any interpretable relationship with the underlying process. It is shown that model selection based solely on the fit to observed data will result in the choice of an unnecessarily complex model that overfits the data, and thus generalizes poorly. The effect of over-fitting must be properly offset by model selection methods. An application example of selection methods using artificial data is also presented.

title: Model Comparisons and Model Selections Based on Generalization Criterion Methodology.
abstract: The purpose of this article is to formalize the generalization criterion method for model comparison. The method has the potential to provide powerful comparisons of complex and nonnested models that may also differ in terms of numbers of parameters. The generalization criterion differs from the better known cross-validation criterion in the following critical procedure. Although both employ a calibration stage to estimate parameters, cross-validation employs a replication sample from the same design for the validation stage, whereas generalization employs a new design for the critical stage. Two examples of the generalization criterion method are presented that demonstrate its usefulness for selecting a model based on sound scientific principles out of a set that also contains models lacking sound scientific principles that are either overly complex or oversimplified. The main advantage of the generalization criterion is its reliance on extrapolations to new conditions. After all, accurate a priori predictions to new conditions are the hallmark of a good scientific theory.

title: Applying Occam’s razor in modeling cognition: A Bayesian approach
abstract: In mathematical modeling of cognition, it is important to have well-justified criteria for choosing among differing explanations (i.e., models) of observed data. This paper introduces a Bayesian model selection approach that formalizes Occam’s razor, choosing the simplest model that describes the data well. The choice of a model is carried out by taking into account not only the traditional model selection criteria (i.e., a model’s fit to the data and the number of parameters) but also the extension of the parameter space, and, most importantly, the functional form of the model (i.e., the way in which the parameters are combined in the model’s equation). An advantage of the approach is that it can be applied to the comparison of non-nested models as well as nested ones. Application examples are presented and implications of the results for evaluating models of cognition are discussed.

title: Information Theory and an Extension of the Maximum Likelihood Principle
abstract: In this paper it is shown that the classical maximum likelihood principle can be considered to be a method of asymptotic realization of an optimum estimate with respect to a very general information theoretic criterion. This observation shows an extension of the principle to provide answers to many practical problems of statistical model fitting.

title: Assessing model mimicry using the parametric bootstrap
abstract: We present a general sampling procedure to quantify model mimicry, defined as the ability of a model to account for data generated by a competing model. This sampling procedure, called the parametric bootstrap cross-fitting method (PBCM; cf. Williams ( (1970) 350; Biometrics 26 (1970) 23)), generates distributions of differences in goodness-of-fit expected under each of the competing models. In the data informed version of the PBCM, the generating models have specific parameter values obtained by fitting the experimental data under consideration. The data informed difference distributions can be compared to the observed difference in goodness-of-fit to allow a quantification of model adequacy. In the data uninformed version of the PBCM, the generating models have a relatively broad range of parameter values based on prior knowledge. Application of both the data informed and the data uninformed PBCM is illustrated with several examples.

title: Parallel and serial processes in visual search.
abstract: A long-standing issue in the study of how people acquire visual information centers around the scheduling and deployment of attentional resources: Is the process serial, or is it parallel? A substantial empirical effort has been dedicated to resolving this issue (e.g., J. M. Wolfe, 1998a, 1998b). However, the results remain largely inconclusive because the methodologies that have historically been used cannot make the necessary distinctions (J. Palmer, 1995; J. T. Townsend, 1972, 1974, 1990). In this article, the authors develop a rigorous procedure for deciding the scheduling problem in visual search by making improvements in both search methodology and data interpretation. The search method, originally used by A. H. C. van der Heijden (1975), generalizes the traditional single-target methodology by permitting multiple targets. Reaction times and error rates from 29 representative search studies were analyzed using Monte Carlo simulation. Parallel and serial models of attention were defined by coupling the appropriate sequential sampling algorithms to realistic constraints on decision making. The authors found that although most searches are conducted by a parallel limited-capacity process, there is a distinguishable search class that is serial.

title: Contraction of time in attention-deficit hyperactivity disorder.
abstract: Attention-deficit hyperactivity disorder (ADHD) has been associated with anomalies in dopamine systems. Recent advances in the understanding of the core cognitive deficits in ADHD suggest that dopamine dysfunction might be expressed through shortened time scales in reward-based learning. Here this perspective is extended by the conjecture that temporal span in working memory systems might generally be shortened. As a test of this conjecture the authors focus on the implicit memory system involved in rhythmic movement, assessing the minimum tempo at which rhythmic feeling can be sustained in adults with diagnosed ADHD and in a control group of normal adults. The authors found that people with ADHD do in fact have a rhythm cut-off that is faster in tempo than those without ADHD. This finding is consistent with the idea that impaired dopamine dynamics have systemic consequences for cognitive function, essentially recalibrating the clock that sets the time scale for the subjective experience of temporal events.

title: Variability in isochronous tapping: higher order dependencies as a function of intertap interval.
abstract: Isochronous serial interval production (ISIP) data, as from unpaced finger tapping, exhibit higher order dependencies (drift). This fact has largely been ignored by the timing literature, one reason probably being that influential timing models assume random variability. Men and women, 22-36 years old, performed a synchronization-continuation task with intertap intervals (ITI) from 0.4 s to 2.2 s. ISIP variability was partitioned into components attributable to drift and 1st-order serial correlation, and the results indicate that (a) drift contributes substantially to the dispersion for longer ITIs, (b) drift and 1st-order correlation are different functions of the ITI, and (c) drift exhibits break close to 1.0 s and 1.4 s ITI. These breaks correspond to qualitative changes in performance for other temporal tasks, which suggests common timing processes across modalities and tasks.