Replicating Empirical Research In Behavioral Ecology: How And Why It Should Be Done But Rarely Ever Is

That empirical evidence is replicable is the foundation of science. Ronald Fisher, a founding father of biostatistics, recommended that a null hypothesis be rejected more than once because “no isolated experiment, however significant in itself, can suffice for the experimental demonstration of any natural phenomenon” (Fisher 1974:14). Despite this demand, animal behaviorists and behavioral ecologists seldom replicate studies. This practice is not part of our scientific culture, as it is in chemistry or physics, due to a number of factors, including a general disdain by journal editors and thesis committees for unoriginal work. I outline why and how we should replicate empirical studies, which studies should be given priority, and then elaborate on why we do not engage in this necessary endeavor. I also explain how to employ various statistics to test the replicability of a series of studies and illustrate these using published studies from the literature.

Related papers:

title: Null hypothesis significance testing: a review of an old and continuing controversy.
abstract: Null hypothesis significance testing (NHST) is arguably the most widely used approach to hypothesis evaluation among behavioral and social scientists. It is also very controversial. A major concern expressed by critics is that such testing is misunderstood by many of those who use it. Several other objections to its use have also been raised. In this article the author reviews and comments on the claimed misunderstandings as well as on other criticisms of the approach, and he notes arguments that have been advanced in support of NHST. Alternatives and supplements to NHST are considered, as are several related recommendations regarding the interpretation of experimental data. The concluding opinion is that NHST is easily misunderstood and misused but that when applied with good judgment it can be an effective aid to the interpretation of experimental data.

title: QUASIREPLICATION AND THE CONTRACT OF ERROR: Lessons from Sex Ratios, Heritabilities and Fluctuating Asymmetry
abstract: ▪ Abstract Selective reporting—e.g., the preferential publication of results that are statistically significant, or consistent with theory or expectation—presents a challenge to meta-analysis and seriously undermines the quest for generalizations. Funnel graphs (scatterplots of effect size vs. sample size) help reveal the extent of selective reporting. They also allow the strength of biological effects to be judged easily, and they reaffirm the value of graphical presentations of data over statistical summaries. Funnel graphs of published results, including: (a) sex-ratio variation in birds, (b) field estimates of heritabilities, and (c) relations between fluctuating asymmetry and individual attractiveness or fitness, suggest selective reporting is widespread and raise doubts about the true magnitude of these phenomena. Quasireplication—the “replication” of previous studies using different species or systems—has almost completely supplanted replicative research in ecology and evolution. Without incentives...

title: Testing and adjusting for publication bias
abstract: How can the scientific literature provide unbiased conclusions if it represents a biased sample of available studies? Publication bias refers to phenomena arising from bias in submitting, reviewing, accepting and publishing scientific results. Direct and indirect methods for investigating publication bias are now readily available, but indirect methods are generally open to alternative interpretations. Publication bias distorts attempts to review a scientific field quantitatively if the likelihood of locating completed studies depends on the strength or direction of the findings of quantitative studies. It is the responsibility of researchers, reviewers and editors to address issues of bias to ensure the existence of an unbiased literature.

title: Statistical significance in psychological research.
abstract: Most theories in the areas of personality, clinical, and social psychology predict no more than the direction of a correlation, group difference, or treatment effect. Since the null hypothesis is never strictly true, such predictions have about a SO-SO chance of being confirmed by experiment when the theory in question is false, since the statistical significance of the result is a function of the sample size. Confirmation of a single directional prediction should usually add little to one's confidence in the theory being tested. Most theories should be tested by multiple corroboration and most empirical generalizations by constructive replication. Statistical significance is perhaps the least important attribute of a good experiment; it is never a sufficient condition for claiming that a theory has been usefully corroborated, that a meaningful empirical fact has been established, or that an experimental report ought to be published. In a recent journal article Sapolsky (1964) developed the following substantive theory: Some psychiatric patients entertain an unconscious belief in the "cloacal theory of birth" which involves the notions of oral impregnation and anal parturition. Such patients should be inclined to manifest eating disorders: compulsive eating in the case of those who wish to get pregnant and anorexia in those who do not. Such patients should also be inclined to see cloacal animals, such as frogs, on the Rorschach. This reasoning led Sapolsky to predict that Rorschach frog responders show

title: Non-significant results in ecology: a burden or a blessing in disguise?
abstract: Null hypothesis significance testing remains a common practice in ecology, despite criticism by statisticians (Yoccoz 1991, Cohen 1994) and numerous suggested alternatives (Jones and Matloff 1986, Fernandez-Duque 1996, Parkhurst 2001). The preoccupation of scientists with the statistical significance of tests as a criterion of study value may lead to the under-reporting of nonsignificant (P 0.05) results in the published literature (the ‘file drawer problem’, Rosenthal 1979). This situation may arise either because non-significant results are not submitted for publication or because they are rejected in the review process. Less severe forms of bias against non-significant results (sometimes referred to as ‘dissemination bias’, Song et al. 2000) include time-lag bias (delayed publication) and place of publication bias (publication in low-circulation journals or in the form of technical reports, conference abstracts or dissertations). As a result, even when published, studies reporting non-significant results may be less accessible to researchers, undercited, and less likely to be indexed in major reference databases – and therefore more likely to go unnoticed. The under-reporting of non-significant results has long been suspected to occur in ecology (Csada et al. 1996), and potential problems caused by publication and dissemination bias against non-significant results for research synthesis, design of ecological experiments and the presentation of the results have been discussed repeatedly in Oikos (Csada et al. 1996, Bauchau 1997, Lortie and Dyer 1999, Kotiaho and Tomkins 2002). However, a recent review by Moller and Jennions (2001) demonstrated that the existing evidence of bias against non-significant results in ecology is based largely on indirect methods (e.g. analysis of the funnel plots or calculation of the fail-safe numbers) which are open to alternative interpretations. Direct evidence of publication bias comes from comparisons of results between unpublished and published studies and from follow-ups of the publication fate of a group of preregistered studies. Several surveys of this kind conducted in medicine have demonstrated that published studies often report a higher effectiveness of treatments than unpublished trials, and that the publication rate of trials providing significant results is considerably higher than those that found no significant differences among treatments (reviewed in Song et al. 2000). Similar analyses of ecological studies have not been undertaken so far because of the logistic problems in obtaining an unbiased sample of unpublished studies. Although some ecological meta-analyses included considerable number of unpublished studies and were able to compare the magnitude of effect size for published and unpublished studies (Thornhill et al. 1999, Jennions et al. 2001), most of the unpublished studies included in the above reviews were very recent, and might therefore be unpublished simply because they had only recently been conducted (Moller and Jennions 2001). Direct evidence of publication bias in ecology is therefore still lacking. In order to find out whether the statistical significance of the results affects the publication of ecological studies, I have followed up the fate of manuscripts from Finnish and Swedish doctoral dissertations on ecological topics. A typical PhD thesis in Finland and Sweden consists of several articles (usually 4–6) and a summary. Some of the papers included in the thesis may already have been published or accepted for publication, while others are in manuscript; these are often submitted and published after the PhD student obtains a doctoral degree. Given that PhD students usually have limited time for completion of their studies and little experience of conducting and writing up research, I assumed that all or at least most of the research conducted during the PhD studies is included in the thesis and, thus, Finnish and Swedish doctoral dissertations provide an unbiased source of completed but as

title: Direct and indirect tests for publication bias: asymmetry and sexual selection
abstract: Recently, several studies have suggested that the biological literature may be biased because studies with particular effect sizes are more likely to get published. However, only one published direct test comparing effect sizes of a representative sample of published and unpublished studies exists in biology. We performed an additional direct test of publication bias, using studies of the relationship between asymmetry and sexual selection. Samples of published and unpublished papers on this relationship were obtained by (1) checking the literature; (2) contacting scientists working on the subject; and (3) posting requests for studies against a reward on Web sites. There was no significant bias in respondents to our request as compared to nonrespondents with respect to effect size or other features of the published studies on the subject. Also, the characteristics of the studies of respondents with and without unpublished research on the subject did not differ significantly. Unpublished and published studies did not differ significantly in effect size adjusted for sample size, using random-effects meta-analysis. The number of unpublished studies we obtained, however, was moderate. Moreover, there was no evidence of a temporal change in mean effect size adjusted for sampling effort between a sample of studies conducted through 1996, a sample of studies conducted thereafter until June 2001, and the sample of unpublished studies in the present study. Indirect tests for publication bias used on the three samples revealed little evidence of publication bias. In conclusion, there is a small to intermediate effect size weighted by sample size of the relationship between asymmetry and sexual selection, and this effect appears to be largely independent of publication bias as revealed by direct and indirect estimates of publication bias.

title: Detecting Publication Bias in Meta‐analyses: A Case Study of Fluctuating Asymmetry and Sexual Selection
abstract: Those familiar with human nature and the publication process acknowledge that biases due to selective reporting of results are likely widespread in all fields of academic inquiry that depend on tools of statistical inference (e.g., see Begg and Berlin 1988; Iyengar and Greenhouse 1988; and the extensive discussion following each). However, although qualitative and quantitative methods exist for assessing the prevalence of selective reporting, and selective reporting has been studied in the medical and social sciences (Begg 1994, and references therein), the issue has received little attention in recent meta-analyses of ecological and evolutionary patterns. Clearly, “if publication bias is present, and if it operates in the same direction for all studies (as is likely), then [meta-analysis] is likely not only to produce biased summary estimates but also to produce estimates which are apparently precise and accurate leading to conclusions which may not only be wrong but appear convincing” (Begg and Berlin 1988, p. 437). To biologists unacquainted with the formal study of publication patterns, the terms “selective reporting” (statistical significance of an outcome influences its likelihood of being reported or published) and “publication bias” (the inflation of average effect size due to selective reporting) may imply a conscious intent to deceive, but this

title: Publication bias in ecology and evolution: an empirical assessment using the ‘trim and fill’ method
abstract: Recent reviews of specific topics, such as the relationship between male attractiveness to females and fluctuating asymmetry or attractiveness and the expression of secondary sexual characters, suggest that publication bias might be a problem in ecology and evolution. In these cases, there is a significant negative correlation between the sample size of published studies and the magnitude or strength of the research findings (formally the ‘effect size’). If all studies that are conducted are equally likely to be published, irrespective of their findings, there should not be a directional relationship between effect size and sample size; only a decrease in the variance in effect size as sample size increases due to a reduction in sampling error. One interpretation of these reports of negative correlations is that studies with small sample sizes and weaker findings (smaller effect sizes) are less likely to be published. If the biological literature is systematically biased this could undermine the attempts of reviewers to summarise actual biology relationships by inflating estimates of average effect sizes. But how common is this problem? And does it really effect the general conclusions of literature reviews? Here, we examine data sets of effect sizes extracted from 40 peer‐reviewed, published meta‐analyses. We estimate how many studies are missing using the newly developed ‘trim and fill’ method. This method uses asymmetry in plots of effect size against sample size (‘funnel plots’) to detect ‘missing’ studies. For random‐effect models of meta‐analysis 38% (15/40) of data sets had a significant number of ‘missing’ studies. After correcting for potential publication bias, 21% (8/38) of weighted mean effects were no longer significantly greater than zero, and 15% (5/34) were no longer statistically robust when we used random‐effects models in a weighted meta‐analysis. The mean correlation between sample size and the magnitude of standardised effect size was also significantly negative (rs=‐0.20, P < 0‐0001). Individual correlations were significantly negative (P < 0.10) in 35% (14/40) of cases. Publication bias may therefore effect the main conclusions of at least 15–21% of meta‐analyses. We suggest that future literature reviews assess the robustness of their main conclusions by correcting for potential publication bias using the ‘trim and fill’ method.

title: The effect of leg band symmetry on female–male association in zebra finches
abstract: Studies have shown that female zebra finches, Taeniopygia guttata, prefer to associate with males wearing symmetric leg band combinations. In this study, I used a four-choice chamber apparatus to investigate this preference, and to determine whether it is influenced by band coloration. I created band combinations by banding males with four leg bands. The symmetrical combinations consisted of two bands on each leg; the asymmetrical combinations consisted of three bands on one leg and one band on the other leg. For each of the symmetry treatments, all four bands were either light green or red. Females did not spend significantly more time associating with males wearing symmetrical bands or with males wearing red bands. The duration of male-female associations was not influenced by an interaction between symmetry category and band colour treatment. None of 10 females spent the most time with males with red symmetrical bands.

title: Female zebra finches prefer symmetrically banded males, but only during interactive mate choice tests
abstract: We used a four-arm choice chamber to investigate whether the symmetry of artificial traits (leg bands) influenced how attractive male zebra finches, Taeniopygia guttata, were to females, and whether a female's preference was influenced by the opportunity to interact with males during choice tests. Females (N=10) were tested separately and moved freely between the chamber's arms, each of which contained a male wearing bands (two per leg) in symmetric, asymmetric or cross-asymmetric configurations. In three of the six tests experienced by each female, one-way glass (mounted diagonally within each arm) prevented males from seeing the female or their own reflection. During the other three tests, transparent glass was used, so males could see and react to the female. A male's band combination had a significant influence on the amount of time that females spent with him, but only during transparent-glass tests, where symmetric males were preferred. The results of our transparent-glass tests were similar to those reported by Swaddle & Cuthill (1994, , in a study where females selected between males that were interacting with their own mirror images (reflected by one-way glass). We discuss why females display the same apparent preference when tested with males engaged in intra- (Swaddle & Cuthill's study) versus intersexual activities (our study). Since females in our study displayed a preference only when males could interact with them, caution may be required when extrapolating the results of static choice tests (e.g. involving models, one-way glass or video images) to situations where the participants can interact.

title: What if there were no significance tests
abstract: Preface. Part I: Overview. L.L. Harlow,  Testing Introduction and Overview. Part II: The Debate: Against and For  Schmidt, J. Hunter, Eight Objections to the Discontinuation of Significance Testing in the Analysis of Research Data. S.A. Mulaik, N.S. Raju, R. Harshman, There Is a Time and Place for Significance Testing. R.P. Abelson, A Retrospective on the Significance Test Ban of 1999 (If There Were No Significance Tests, They Would Be Invented). Part III: Suggested Alternatives to Significance Testing. R.J. Harris, Reforming Significance Testing via Three-Valued Logic. J.S. Rossi, Spontaneous Recovery of Verbal Learning: A Case Study in the Failure of Psychology as a Cumulative Science. J.H. Steiger, R.T. Fouladi, Noncentrality Interval Estimation and the Evaluation of Statistical Models. R.P. McDonald, Goodness of Approximation in the Linear Model. Part IV: A Bayesian Approach to Hypothesis Testing. R.M. Pruzek, An Introduction to Bayesian Inference and Its Application. D. Rindskopf, Testing 'Small,' Not Null, Hypotheses: Classical and Bayesian Approaches. C.S. Reichardt, H.F. Gollob, When Confidence Intervals Should Be Used Instead of Statistical Significance Tests, and Vice Versa. Part V: Philosophy of Science Issues. W.W. Rozeboom, Good Science Is Abductive, Not Hypothetico-Deductive. P.E. Meehl, The Problem Is Epistemology, Not  Replace

title: Extracting a maximum of useful information from statistical research data.
abstract: The practice of statistical inference in psychological research is critically reviewed. Particular emphasis is put on the fast pace of change from the sole reliance on null hypothesis significance testing (NHST) to the inclusion of effect size estimates, confidence intervals, and an interest in the Bayesian approach. We conclude that these developments are helpful for psychologists seeking to extract a maximum of useful information from statistical research data, and that seven decades of criticism against NHST is finally having an effect.

title: Are significance thresholds appropriate for the study of animal behaviour?
abstract: Recent papers in Animal Behaviour concerning statistical power analysis have generated an interesting discussion concerning how the results of a significance test should be interpreted (Greenwood 1993; Thompson & Neill 1993; Johnsson 1996; Thomas & Juanes 1996). These discussions are welcome and overdue, and have surely improved the quality of manuscripts submitted to Animal Behaviour. I believe, however, that there is still room for improvement, and that this improvement should be relatively painless to authors. This improvement depends upon a basic understanding of the relationship between statistical power, the strength of the biological association or effect present, and P values, and what this means for the way in which the results of a significance test are interpreted. Power is defined as the probability of rejecting a false null hypothesis and depends upon three related variables: the sample size (N), the magnitude of the effect or association in the population (i.e. the size of the differences between group means, and the variance associated with these differences, or the strength of the correlation), often referred to as the effect size, and the ‘significance criterion’ (Cohen 1992). The significance criterion here refers to the preset type I error rate (alpha), or what Animal Behaviour’s ‘Instructions to Authors’ refer to as a significance threshold. The relationship between power, sample size, effect size and significance threshold implies several things. Because power increases with sample size, for a given effect, larger sample sizes will generally result in lower P values. Thus, if sample sizes for similar experiments are different, P values will differ even if both experiments have detected the same effect. Similarly, identical P values will rarely mean identical effect sizes. Additionally, power increases as measurement error decreases (something often not mentioned in discussions of power) so that two experiments that both result in a mean difference between experimental and control groups of (say) 10 g may result in very different P values if one study was conducted in the field on wild animals and another conducted under semicontrolled conditions with captive animals (Hurlbert 1994). What this means is that stating that the results of one study are statistically significant while another’s are nonsignificant, ‘moderately significant’, or ‘highly significant’ says very little about the biological conclusions one should draw from these experiments. Understanding the relationship between power and sample size, and thus P values, is important. However, power is always defined as the probability of getting a significant result at a prespecified significance threshold, traditionally set at 0.05. Animal Behaviour’s ‘Instructions to Authors’ state that ‘departure from a significance threshold of 0.05 should be stated and justified . . .’. I would like to argue that in the field of animal behaviour, preset significance thresholds and the accept/reject decisions that accompany them are not logical, and should perhaps be abandoned. This does not mean that I advocate eliminating significance tests, as I will explain, but rather that perhaps there are more reasonable ways to use them in the study of animal behaviour. My primary purpose in writing this commentary is not to suggest that what we are currently doing in the field is bad science, but simply that there is room for improvement.

title: In support of null hypothesis significance testing
abstract: Many criticisms have been levelled at null hypothesis significance testing (NHST). It is argued here that although there is reason to doubt that data subjected only to NHST have been subjected to sufficient analysis, the search for clear answers to well–formulated questions derived from substantive hypotheses is well served by NHST. To reliably draw inferences from data, however, NHST may need to be complemented by additional methods of analysis, such as the use of confidence intervals and of estimates of the degree of association between independent and dependent variables. It is argued that these should be seen as complements of, rather than as substitutes for, NHST since they do not directly test the strength of evidence against a null hypothesis.

title: What Statistical Significance Testing Is, and What It Is Not
abstract: AbstractA test of statistical significance addresses the question, How likely is a result, assuming the null hypotheses to be true. Randomness, a central assumption underlying commonly used tests of statistical significance, is rarely attained, and the effects of its absence rarely acknowledged. Statistical significance does not speak to the probability that the null hypothesis or an alternative hypothesis is true or false, to the probability that a result would be replicated, or to treatment effects, nor is it a valid indicator of the magnitude or the importance of a result. The persistence of statistical significance testing is due to many subtle factors. Journal editors are not to blame, but as publishing gatekeepers they could diminish its dysfunctional use.

title: Statistical Procedures and the Justification of Knowledge in Psychological Science
abstract: Justification, in the vernacular language of philosophy of science, refers to the evaluation, defense, and confirmation of claims of truth. In this article, we examine some aspects of the rhetoric of justification, which in part draws on statistical data analysis to shore up facts and inductive inferences. There are a number of problems of methodological spirit and substance that in the past have been resistant to attempts to correct them. The major problems are discussed, and readers are reminded of ways to clear away these obstacles to justification.

title: Null hypothesis significance testing. On the survival of a flawed method.
abstract: Null hypothesis significance testing (NHST) is the researcher's workhorse for making inductive inferences. This method has often been challenged, has occasionally been defended, and has persistently been used through most of the history of scientific psychology. This article reviews both the criticisms of NHST and the arguments brought to its defense. The review shows that the criticisms address the logical validity of inferences arising from NHST, whereas the defenses stress the pragmatic value of these inferences. The author suggests that both critics and apologists implicitly rely on Bayesian assumptions. When these assumptions are made explicit, the primary challenge for NHST--and any system of induction--can be confronted. The challenge is to find a solution to the question of replicability.

title: The Data Analysis Dilemma: Ban or Abandon. A Review of Null Hypothesis Significance Testing.
abstract: Null Hypothesis Significance Testing (NHST) is reviewed in a historical context. The most vocal criticisms of NHST that have appeared in the literature over the past 50 years are outlined. The authors conclude, based on the criticism of NHST and the alternative methods that have been proposed, that viable alternatives to NHST are currently available. The use of effect magnitude measures with surrounding confidence intervals and indications of the reliability of the study are recommended for individual research studies. Advances in the use of meta-analytic techniques provide us with opportunities to advance cumulative knowledge, and all research should be aimed at this goal. The authors provide discussions and references to more information on effect magnitude measures, replication techniques and meta-analytic techniques. A brief situational assessment of the research landscape and strategies for change are offered.

title: AERA Editorial Policies Regarding Statistical Significance Testing: Three Suggested Reforms.
abstract: The present comment reviews practices revolving around tests of statistical significance. First, the logic of statistical significance testing is presented in an accessible manner; many people who use statistical tests might not place such a premium on the tests if these individuals understood what the tests really do, and what the tests do not do. Second, the etiology of decades of misuse of statistical tests is briefly explored; we must understand the bad implicit logic of persons who misuse statistical tests if we are to have any hope of persuading them to alter their practices. Third, three revised editorial policies that would improve conventional practice are highlighted.

title: What If There Were No More Bickering About Statistical Significance Tests
abstract: Questions and concerns are directed to those who advocate replacing statistical hypothesis testing with alternative data-analysis strategies. It is further suggested that: (1) commonly recommended hypothesis-testing alternatives are anything but perfect, especially when allowed to stand alone without an accompanying inferential filtering device; (2) various hypothesis-testing modifications can be implemented to make the hypothesis-testing process and its associated conclusions more credible; and (3) hypothesis testing, when implemented intelligently, adds importantly to the story-telling function of a published empirical research investigation.

title: A Primer of Ecology
abstract: With the aim of teaching students the essential models in population and community ecology, this book explains in detail the basic concepts of exponential and logistic population growth, age-structured demography, metapopulation dynamics, competition, predation and island biogeography. Each chapter is graded from simple material to advanced material, for both beginning undergraduates and upper-level undergraduates and postgraduates.

title: Sperm competition, male prudence and sperm-limited females
abstract: Sperm are produced in astronomical numbers compared with eggs, and there is good evidence that sperm competition is the force behind the evolution of many tiny sperm. However, sperm production inevitably has costs. Recent research shows that male ejaculate expenditure is dynamic in both time and space, and that males are sensitive to risks of sperm competition and can vary ejaculate size accordingly. We focus on studies showing that males assess mating status and relative fecundity of females, and reveal that modulation of ejaculate investment by males can sometimes result in sperm limitation for females.

title: Is Sperm Cheap? Limited Male Fertility and Female Choice in the Lemon Tetra (Pisces, Characidae)
abstract: In the laboratory, fertilization rates achieved by male lemon tetras decline with spawning frequency. Even when the number of females is not limited, males can produce only four times as many offspring as females. Females show a preference for males that have not recently spawned as opposed to those that have. The cost of producing sufficient sperm to maximize fertilization rates may therefore reduce the intensity of sexual selection in this polygamous fish species.

title: Geographic variation in behavior : perspectives on evolutionary mechanisms
abstract: The Use of Behavioral Ecotypes in the Study of Evolutionary Processes 2. Different Spatial Scales of Natural Selection and Gene Flow: The Evolution of Behavioral Geographic Variation and Phenotypic Plasticity 3. The Evolution of Behavioral Norms of Reaction as a Problem in Ecological Genetics: Theory, Methods, and Data 4. Geographic Variations on Methodological Themes in Comparative Ethology: A Natricine Snake Perspective 5. Geographic Variation in Behavior of a Primate Taxon: Stress Responses as a Proximate Mechanism in the Evolution of Social Behavior 6. Ecology, Phenotype, and Character Evolution in Bluegill Sunfish: A Population Comparative Approach 7. The Causes and Consequences of Variation in Antipredator Behavior: Perspectives from Fish Populations 8. Geographic Variation and the Microevolution of Avian Migratory Behavior 9. Effects of Relaxed Natural Selection on the Evolution of Behavior 10. Variation in Advertisment Calls of Anurans Across Zonal Interactions: The Evolution and Breakdown of Homogamy 11. Geographic Variation in Animal Communication Systems 12. Geographic Variation  Sex, Signals, and Speciation 13. Thoughts on Geographical Variation in Behavior

title: Natural Selection of Parental Ability to Vary the Sex Ratio of Offspring
abstract: Theory and data suggest that a male in good condition at the end of the period of parental investment is expected to outreproduce a sister in similar condition, while she is expected to outreproduce him if both are in poor condition. Accordingly, natural selection should favor parental ability to adjust the sex ratio of offspring produced according to parental ability to invest. Data from mammals support the model: As maternal condition declines, the adult female tends to produce a lower ratio of males to females.

title: Deer destiny determined by density
abstract: Theory predicts that socially dominant females should produce more sons than daughters. But when times are hard, the higher growth rate of males means that proportionately more will die during development. New evidence shows that, in red deer, both of these effects operate in concert.

title: Intra-sexual selection in Drosophila
abstract: SINCE Darwin first wrote on the subject in 1871, sexual selection has been generally accepted as one of the basic facts of biology. The evidence in its favour seems, however, to be mainly circumstantial. Its existence has usually been inferred from sex differences depending on what are called secondary sexual characters which are supposed to have arisen as results of that selection. Such an approach has its dangers, and Huxley (1938) has made important criticisms of the original concept of sexual selection. He has shown that a large number of characters which have been attributed to sexual selection are unconnected with competition for mates. This is particularly the case in monogamous birds which offer some of the most striking examples of secondary sexual differences. In the first place monogamy, at least when the sexes are numerically equal, is the mating system least likely to develop sexual selection. In the second place, and more important, observations on bird behaviour have shown that much of the display of birds occurs after pairing, when competition must have ceased. Such sexual differences are concerned, either with inducing the female to copulate, or with maintaining the association of the sexes as long as it is necessary for the rearing of the young. Huxley therefore introduced the term epigamic to apply to characters which increased the fertility of a given mating and therefore had a selective value for the species as a whole. Epigamic selection includes the major part of what Darwin meant by sexual selection. It also includes selection for characters to which Darwin did not refer, such as the structure of copulatory organs, sex differences in frequency of crossing over, and the XY mechanism. It is only a special case of natural selection as generally understood. What remains of Darwinian sexual selection has been called intra-sexual selection, which denotes that it involves competition between members of one sex for mates. It can only indirectly affect the survival of the species and then is often deleterious (e.g. the cumbersome antlers of the stag). There is not invariably, however, a clear distinction between epigamic and intrasexual selection. In a promiscuous species like Drosophila pairing and copulation are synchronous. Courtship behaviour determines the number of mates and therefore enters into intra-sexual selection.

title: Chance can produce a sex difference in variance in mating success and explain Bateman's data
abstract: In many species the sexes may differ in their contribution to the production and care of offspring. Previous theory and observations have shown that the sex which contributes least will also be the sex with the highest variance in mating success. I show that this phenomenon can result from random mating. Bateman's (1948) experiment showed that male Drosophila melanogaster were more variable in their mating success than were females. This additional variability is usually attributed to intra-male competition and female choice but I show that this phenomenon can result from random mating.

title: Animal Mating Systems: A Synthesis Based on Selection Theory
abstract: Following principles used by A. J. Bateman, we identify the relationship between fecundity and mating success as the central feature in the operation of mating systems. Using selection theory from the field of quantitative genetics, we define the sexual selection gradient as the average slope of the relationship between fecundity and mating success and show how it can be estimated from data. We argue that sexual selection gradients are the key to understanding how the intensity of sexual selection is affected by mate provisioning, parental investment, and sex ratio.

title: Promiscuity: An Evolutionary History of Sperm Competition
abstract: Males are promiscuous and ferociously competitive. Females--both human and of other species--are naturally monogamous. That at least is what the study of sexual behavior after Darwin assumed, perhaps because it was written by men. Only in recent years has this version of events been challenged. Females, it has become clear, are remarkably promiscuous and have evolved an astonishing array of strategies, employed both before and after copulation, to determine exactly who will father their offspring. Tim Birkhead reveals a wonderful world in which males and females vie with each other as they strive to maximize their reproductive success. Both sexes have evolved staggeringly sophisticated ways to get what they want--often at the expense of the other. He introduces us to fish whose first encounter locks them together for life in a perpetual sexual embrace; hermaphrodites who joust with their reproductive organs, each trying to inseminate the other without being inseminated; and tiny flies whose seminal fluid is so toxic that it not only destroys the sperm of rival males but eventually kills the female. He explores the long and tortuous road leading to our current state of knowledge, from Aristotle's observations on chickens, to the first successful artificial insemination in the seventeenth century, to today's ingenious molecular markers for assigning paternity. And he shows how much human behavior--from the wife-sharing habits of Inuit hunters to Charlie Chaplin's paternity case--is influenced by sperm competition. Lucidly written and lavishly illustrated, with a wealth of fascinating detail and vivid examples, Promiscuity is the ultimate guide to the battle of the sexes.

title: The Problem with Paradigms: Bateman's Worldview as a Case Study1
abstract: Bateman's principles, their corollaries and predictions constitute a paradigm for the study of sexual selection theory, evolution of mating systems, parental investment theory, and sexual dimorphism in male and female behavior. Some aspects of this paradigm have been challenged in recent years, while others have been supported by empirical and theoretical research. We re-examine Bateman's 1948 paper in detail, including some methodological problems. Additionally, we review three areas in which an over-reliance on Bateman's predictions about sexual dynamics hindered our ability to understand the potential importance of certain behaviors: 1) male mate choice and sperm allocation; 2) the role of females in initiating and soliciting extra-pair copulations and fertilizations; and 3) the role of females in lekking systems, in which recent evidence suggests that copulations with multiple males (polyandrous behavior) may be common. We conclude this introduction to the symposium by emphasizing the heuristic value of Bateman's contributions, as well as the problems that arise when Bateman's paradigm is viewed through the lens of modern behavioral ecology and evolutionary theory.

title: Bilateral Symmetry and Sexual Selection: A Meta‐Analysis
abstract: A considerable body of primary research has accumulated over the last 10 yr testing the relationship between developmental instability in the form of fluctuating asymmetry and performance of individuals in mating success itself or sexual attractiveness. This research comprises 146 samples from 65 studies of 42 species of four major taxa. We present the results of a meta‐analysis of these studies, which demonstrates that there is indeed an overall significant, moderate negative relationship: for studies, the overall mean Pearson's r or effect size = −.42, P <.0005; for species, the overall mean r = −.34, .01 < P < .025. Based on calculated fail‐safe numbers, the effect‐size estimates are highly robust against any publication or reporting bias that may exist. There is considerable evidence that the magnitude of the negative correlation between fluctuating asymmetry and success related to sexual selection is greater for males than for females, when a secondary sexual trait rather than an ordinary trait is studied, with experimentation compared with observation, and for traits not involved with mobility compared with traits affecting mobility. There is also limited evidence that higher taxa may differ in effect size and that intensity of sexual selection negatively correlates with effect size.

title: The Biological Significance of Fluctuating Asymmetry and Sexual Selection: A Reply to Palmer
abstract: Meta-analysis is a statistical technique used to draw inferences from and summarize large empirical literatures, one that has become increasingly prevalent in the fields of ecology and evolution in recent years (Gurevitch et al. 1992; Gurevitch and Hedges 1993 and in press; Arnqvist and Wooster 1995; Arnqvist et al. 1996; Moller and Thornhill 1997). As noted by a number of authors (e.g., Begg 1994; Gurevitch and Hedges, in press) and most recently by Palmer (1999, in this issue), however, meta-analytic summaries do not necessarily result in unbiased conclusions. For instance, if a research literature has been affected by systematic selective reporting and thus publication bias, the mean effect size in the literature may be overestimated. Following others (Light and Pillemer 1984; Begg 1994), Palmer (1999, in this issue) recommends graphic means of examining selective reporting. We endorse these recommendations and second Palmer’s call for caution in interpreting meta-analytic results. Palmer (1999, in this issue) illustrates the detection of selective reporting by examining a particular meta-analysis, one conducted by Moller and Thornhill (1998) on sexual selection and fluctuating asymmetry (FA). The FA is purported to tap into individual variation in developmental instability. Sexual selection has been thought to favor individuals who possess low developmental insta-

title: Fluctuating paradigm
abstract: A prominent paradigm in evolutionary biology over the last ten years has been the role of fluctuating asymmetry in sexual selection. Fluctuating asymmetry in bilaterally paired traits, and in particular sexual traits, has been proposed to be a reliable indicator of individual quality and the focus of selection through sexual competition and attractiveness. We surveyed the literature on fluctuating asymmetry and sexual selection and found a marked chronological decline in the proportion of studies supporting the paradigm through the 1990s. Our data are supported by an independent meta–analysis of the literature. The data conform with the notion of a scientific revolution in which the early phase of a paradigm change is characterized by a publication bias, a less critical approach to research, or both. The patterns we observe in the fluctuating asymmetry literature suggest caution in drawing general conclusions from meta–analyses conducted before revolutions have settled.

title: Fluctuating Asymmetry, Animal Behavior, and Evolution
abstract: Publisher Summary The chapter introduces fluctuating asymmetry and shows why it is relevant to behavioral ecologists. It illustrates where the knowledge about fluctuating asymmetry is lacking in important areas, and to offer suggestions that will help behavioral ecologists fill those gaps. The chapter reviews relationships between asymmetry and fitness. These relationships appear to be taxon and trait specific. The specificity of such relationships probably arises from the weak correlation of fluctuating asymmetry with developmental instability. The chapter discusses common methodological problems of studying fluctuating asymmetry. A hypothesis is presented that fluctuating asymmetry may have significant genetic redundancy and could be viewed as a neutral character. Many behavioral ecologists have quantified fluctuating asymmetry in secondary sexual characters and investigated whether such asymmetry plays a role in sexual selection processes. The chapter concludes that a link is established between behavioral studies of fluctuating asymmetry and broader evolutionary questions, and it is suggested that researchers adopt a more integrated approach that brings together a proximate understanding of the impact of asymmetry with more long-term studies of the origins of asymmetry and evolutionary consequences of selection against fluctuating asymmetry.

title: Manipulation of host behaviour by parasites: a weakening paradigm?
abstract: New scientific paradigms often generate an early wave of enthusiasm among researchers and a barrage of studies seeking to validate or refute the newly proposed idea. All else being equal, the strength and direction of the empirical evidence being published should not change over time, allowing one to assess the generality of the paradigm based on the gradual accumulation of evidence. Here, I examine the relationship between the magnitude of published quantitative estimates of parasite–induced changes in host behaviour and year of publication from the time the adaptive host manipulation hypothesis was first proposed. Two independent data sets were used, both originally gathered for other purposes. First, across 137 comparisons between the behaviour of infected and uninfected hosts, the estimated relative influence of parasites correlated negatively with year of publication. This effect was contingent upon the transmission mode of the parasites studied. The negative relationship was very strong among studies of parasites which benefit from host manipulation (transmission to the next host occurs by predation on an infected intermediate host), i.e. among studies which were explicit tests of the adaptive manipulation hypothesis. There was no correlation with year of publication among studies on other types of parasites which do not seem to receive benefits from host manipulation. Second, among 14 estimates of the relative, parasite–mediated increase in transmission rate (i.e. increases in predation rates by definitive hosts on intermediate hosts), the estimated influence of parasites again correlated negatively with year of publication. These results have several possible explanations, but tend to suggest biases with regard to what results are published through time as accepted paradigms changed.

title: Of moths and men : an evolutionary tale : intrigue, tragedy & the peppered moth
abstract: The tale of a flagrant scientific fraud, its cover-up and the scientific incompetence behind the most important paradigm in evolutionary biology: Charles Darwin's theory of evolution. The fraud began in 1953, with H.B.D. Kettlewell, an amateur but charismatic lepidoptorist attached to Oxford University. Using studies of the peppered moth (Biston betularia) he showed the process of natural selection at work over a period of months rather than millennia. The naturally light-coloured moth was found, in Kettlewell's experiments, to have mutated to a darker variety (Biston carbonaria) in industrial areas where the darker colour would prove a more effective camouflage against predator birds. The conclusive evidence was drawn from the rate at which light and dark moths appeared on the trunks of an industrially influenced forest and the rate at which the lighter, poorly camouflaged moths were consumed by birds. There were only two problems: no one was sure that birds naturally ate the pepper moth, and the reason they were consumed in such apparently conclusive numbers was that they had been glued to the trees by Kettlewell. It was not the evolutionary hypothesis that was nailed but the moth. Overturning such a transparent fraud which nonetheless supported the most important tenet of evolutionary biology fell to rival lepidoptorist Theodore Sargent. The British scientific establishment ranged itself against him, bullied and slandered him. Here is Darwinism not in the abstract but rooted in the world of competitve science and scientific instiutions: it's a tale of bad science. In the survival of the fittest, truth sometimes finishes second.

title: Interpersonal expectancy effects: the first 345 studies
abstract: The research area of interpersonal expectancy effects originally derived from a general consideration of the effects of experimenters on the results of their research. One of these is the expectancy effect, the tendency for experimenters to obtain results they expect, not simply because they have correctly anticipated nature's response but rather because they have helped to shape that response through their expectations. When behavioral researchers expect certain results from their human (or animal) subjects they appear unwittingly to treat them in such a way as to increase the probability that they will respond as expected. In the first few years of research on this problem of the interpersonal (or interorganism) self-fulfilling prophecy, the “prophet” was always an experimenter and the affected phenomenon was always the behavior of an experimental subject. In more recent years, however, the research has been extended from experimenters to teachers, employers, and therapists whose expectations for their pupils, employees, and patients might also come to serve as interpersonal self-fulfilling prophecies. Our general purpose is to summarize the results of 345 experiments investigating interpersonal expectancy effects. These studies fall into eight broad categories of research: reaction time, inkblot tests, animal learning, laboratory interviews, psychophysical judgments, learning and ability, person perception, and everyday life situations. For the entire sample of studies, as well as for each specific research area, we (1) determine the overall probability that interpersonal expectancy effects do in fact occur, (2) estimate their average magnitude so as to evaluate their substantive and methodological importance, and (3) illustrate some methods that may be useful to others wishing to summarize quantitatively entire bodies of research (a practice that is, happily, on the increase).

title: Improving What Is Published A Model in Search of an Editor
abstract: Researchers and practitioners of psychology have expressed considerable dissatisfaction with much of what is published in professional journals. Three major areas of discontent are as follows: (a) Many articles focus on irrelevant topics; (b) the use of statistical significance testing often results in meaningless or unusable findings; and (c) the decision-making process for manuscript acceptance/rejection may be biased. Each of these issues is discussed, and an alternative model for manuscript submission is proposed. The advantages and limitations of this model are presented as related to the three areas of current dissatisfaction. Much dissatisfaction has been expressed regarding the quality of published articles and the decision-making process in manuscript acceptance and rejection. The purpose of this article is to review three major areas of discontent and to propose a model of manuscript submission and review that may greatly mitigate the current malaise.l The three major concerns regarding publication practice to be discussed are relevancy, meaningfulness, and bias. T h e I s s u e o f Re l evancy Lindsey (1977) asked, "How is it that so much triviality, illiteracy, and dullness is yearly entered into the scientific publication stream?" (p. 579). Richard Nisbett's (1978) recommendations to psychologists provide a partial answer to Lindsey's question. Nisbett advised researchers interested in increasing the chances of publishing their empirical investigations to avoid creative or innovative experimental designs and to concentrate efforts on areas that are easy to test and are noncontroversial. Unobtrusive, circumstantial data suggest that disenchantment exists throughout the profession of psychology with what appears in journal articles. A survey conducted by Garvey and Griffith (1971) indicated that for any given study, only about 200 psychologists will read its contents within the first 60 days it appears in print. In the absence of comparable data from other scientific disciplines, it is difficult to determine if this finding indicates that psychologists' interest in reading research is representative of most scientists, or if psychologists find few studies worthy of attention. Garfield's (1972) data relate to this issue. He formulated an "impact factor" (i.e., average citation noted per published article, after correction for the number of articles a journal publishes yearly) for the 152 most frequently cited journals in science and technology. The two most often cited journals in psychology are  Bulletin (ranked 35th and 50th, respectively). However, the Psychological Review mainly contains theoretical articles, and Psychological Bulletin consists primarily of reviews of research (Markle & Rinn, 1977). The most frequently cited psychological journal of an experimental nature (ranked 117th), is the Journal of Experimental Analysis of Behavior (Markle & Rinn, 1977). This journal's impact factor is 2.3, suggesting that even articles from the most often cited experimental journal in psychology are often viewed as inconsequential and unlikely to be referenced in articles published in other journals. Garfield's (1972) results have led one reviewer (of this article) to conclude, "One would like to think that if scientists were content with what is currently being published, they would pay more attention to it (cite it) when they write their own papers." (Anonymous personal communication, September 11, 1987). The above data suggest that much of psychology does not advance by an accumulative progression of empirically verifiable facts, but rather, that many investigators conduct isolated studies that rarely aid colleagues in this effort to understand psychological phenomena. Results of surveys exploring practitioners' concerns with publication practices have yielded uniform results over the years. Practitioners report that psychotherapy research has little value for clinical application. When psychologists are requested to rank order the usefulness of informational sources to their practice, research articles and books of empirical research are consistently rated at the bottom on the scale (Cohen, 1979; Cohen, Sargent, & Sechrest, 1986; Morrow-Bradley & Elliot, 1986). Morrow-Bradley and Elliot's (1986) review of 18 references concludes, "With virtual unanimity, psychotherapy researchers have argued that (a) psychotherapy research should yield information useful to practicing therapists, (b) such research to date has not done so, and (c) this problem should be remedied" (p. 188). The end result is that many forms of therapy are adopted before data demCorrespondence concerning this article should be addressed to Joel Kupfersmid, 1560 Callander, Hudson, OH 44236. I The three areas of dissatisfaction are not intended to be an exhaustive list. Rather, these issues represent areas where dissatisfaction can be reduced if the proposed model is instituted. August 1988 9 American Psychologist Copyright 1988 by the American Psychological Association, Inc. 0003-066X/88/$00.75 Vol. 43, No. 8, 635-642 635 onstrating effectiveness are available (Barlow, Hayes, & Nelson, 1984). When survey respondents are asked to list the specific aspects of psychotherapy research that contribute most to their dissatisfaction, the relevancy of topics addressed, the manner in which hypotheses are tested, and the inappropriateness of statistical significance testing emerge as prime areas of discontent. No attitude survey was found that directly asked researchers and academicians their opinion concerning the relevancy of published studies in psychology. Given the above information regarding the opinion and behaviors of psychologists in general and the attitudes expressed by practitioners, it would not be surprising if experimentalists shared many of the same concerns. The Concern for Meaningfulness The paradigm for most experimental and correlational studies consists of postulating no difference (null hypothesis) between groups or variables, a hypothesis that the researcher then attempts to refute. Statistical significance testing is the method most experimenters adopt to determine whether the null hypothesis may be confidently rejected or retained. Significance testing involves selecting a level of probability (p value) to determine "how improbable an event could be under the null hypothesis" (Bakan, 1966, p. 429). Usually a probability of 5% or less (p < .05) is selected. "Thus the p value may be used to make a decision about accepting or rejecting the idea that chance caused the results. This is what statistical significance testing is--nothing more, nothing less" (Carver, 1978, p. 387). The first problem is that the no difference (null) hypothesis is never capable of being retained; that is, the null hypothesis is always false to begin with (Bakan, 1966; Greenwald, 1975; Meehl, 1978). " I f by the null hypothesis one refers to the hypothesis of exactly no difference or exactly no correlation, and so forth, then the initial probability of the null hypothesis being true must be regarded effectively as zero" (Greenwald, 1975, p. 6). Lykken (1968/1970) added that it is "foolish" even to suppose that the difference between two groups, or the correlation between two variables, is ever zero. Researchers can confidently make this claim because no two groups of subjects or variables are ever effectively equal; rather, the null hypothesis will always be rejected tfthe experimenter has a large enough sample N. Meehl (1978) concluded that "reliance on merely refuting the null h y p o t h e s i s . . , is a terrible mistake, is basically unsound, poor scientific strategy, and one of the worst things that ever happened in the history of psychology" (p. 817). A second problem related to the meaningfulness of "statistically significant" findings is that what is significant statistically and what is "significant" in a meaningful sense may be contradictory. The question is never whether the result is statistically significant, but rather, at what N the data would reach statistical significance. Clinicians have expressed dissatisfaction with the use of statistical significance testing because the procedure bears no relationship to the size effect (i.e., degree of difference between two samples and/or degree of correlation; Smith, Glass, & Miller, 1980). The word "significance" is used even for trivial findings. Furthermore, statistical significance is a measure of group effects, whereas the practitioner is concerned with individuals (Barlow et at., 1984). Fifty-two percent of psychotherapists surveyed (Morrow-Bradley & Elliot, 1986) criticized the use of statistical significance testing because the information does not address the important question of how many subjects changed or to what degree. Quoting Bergin and Strupp (1972, p. 440), Barlow et al. (1984) noted, In the area of psychotherapy, the kinds of effects we need to demonstrate . . , should be significant enough so that they are readily observable by inspection or descriptive statistics. If this cannot be done, no fixation upon statistical and mathematical niceties will generate fruitful insights.  Furthermore, in all of experimentation, it is critical to collect data that are relevant to psychological inquiry such that, regardless of outcome, via statistical manipulation, meaningful results are generated. Jacobson, Follette, and Revenstorf (1984) directly addressed this issue by eschewing traditional statistical significance tests for psychotherapy outcome research and argued in favor of a "clinical significance" test. The authors provided a summary of five such measures as well as presenting their own approach: "Therefore, we propose that a change in therapy is clinically significant when the client moves from the dysfunctional to the functional range during the course of therapy on whatever variable is being used to measure the clinical problem"