Cycle frequency in standard Rock-Paper-Scissors games: Evidence from experimental economics

The Rock–Paper–Scissors (RPS) game is a widely used model system in game theory. Evolutionary game theory predicts the existence of persistent cycles in the evolutionary trajectories of the RPS game, but experimental evidence has remained to be rather weak. In this work, we performed laboratory experiments on the RPS game and analyzed the social-state evolutionary trajectories of twelve populations of N=6 players. We found strong evidence supporting the existence of persistent cycles. The mean cycling frequency was measured to be 0.029±0.009 period per experimental round. Our experimental observations can be quantitatively explained by a simple non-equilibrium model, namely the discrete-time logit dynamical process with a noise parameter. Our work therefore favors the evolutionary game theory over the classical game theory for describing the dynamical behavior of the RPS game.

Related papers:

title: Evolutionary game theory: Theoretical concepts and applications to microbial communities
abstract: Ecological systems are complex assemblies of large numbers of individuals, interacting competitively under multifaceted environmental conditions. Recent studies using microbial laboratory communities have revealed some of the self-organization principles underneath the complexity of these systems. A major role of the inherent stochasticity of its dynamics and the spatial segregation of different interacting species into distinct patterns has thereby been established. It ensures the viability of microbial colonies by allowing for species diversity, cooperative behavior and other kinds of “social” behavior. 
 
A synthesis of evolutionary game theory, nonlinear dynamics, and the theory of stochastic processes provides the mathematical tools and a conceptual framework for a deeper understanding of these ecological systems. We give an introduction into the modern formulation of these theories and illustrate their effectiveness focussing on selected examples of microbial systems. Intrinsic fluctuations, stemming from the discreteness of individuals, are ubiquitous, and can have an important impact on the stability of ecosystems. In the absence of speciation, extinction of species is unavoidable. It may, however, take very long times. We provide a general concept for defining survival and extinction on ecological time-scales. Spatial degrees of freedom come with a certain mobility of individuals. When the latter is sufficiently high, bacterial community structures can be understood through mapping individual-based models, in a continuum approach, onto stochastic partial differential equations. These allow progress using methods of nonlinear dynamics such as bifurcation analysis and invariant manifolds. We conclude with a perspective on the current challenges in quantifying bacterial pattern formation, and how this might have an impact on fundamental research in non-equilibrium physics.

title: Evolutionary economics goes mainstream: A review of the theory of learning in games
abstract: The Theory of Learning in Games by Fudenberg and Levine surveys a key branch of evolutionary economics from a mainstream perspective. Its publication provides an opportunity to reassess the prospects and goals for evolutionary economics.

title: Theory of Games and Economic Behavior.
abstract: This is the classic work upon which modern-day game theory is based. What began more than sixty years ago as a modest proposal that a mathematician and an economist write a short paper together blossomed, in 1944, when Princeton University Press published "Theory of Games and Economic Behavior." In it, John von Neumann and Oskar Morgenstern conceived a groundbreaking mathematical theory of economic and social organization, based on a theory of games of strategy. Not only would this revolutionize economics, but the entirely new field of scientific inquiry it yielded--game theory--has since been widely used to analyze a host of real-world phenomena from arms races to optimal policy choices of presidential candidates, from vaccination policy to major league baseball salary negotiations. And it is today established throughout both the social sciences and a wide range of other sciences.

title: Evolutionary game theory
abstract: It is often the case that the best thing for an animal or plant to do depends on what other members of the population are doing. In more technical language, the fitnesses of different phenotypes are frequency-dependent. Evolutionary game theory has been developed to analyse such cases. In this paper I aim to do three things. First, I describe the concepts of evolutionary game theory in the context of a simple game, the Hawk-Dove game, and compare them with the concepts of classical game theory originating with Von Neumann and Morgenstern (1953) . Second, I list some of the applications of the theory. Finally, I suggest how the theory can tell us something about the evolution of learning.

title: Population Games And Evolutionary Dynamics
abstract: This text offers a systematic, rigorous, and unified presentation of evolutionary game theory, covering the core developments of the theory from its inception in biology in the 1970s through recent advances. Evolutionary game theory, which studies the behavior of large populations of strategically interacting agents, is used by economists to make predictions in settings where traditional assumptions about agents' rationality and knowledge may not be justified. Recently, computer scientists, transportation scientists, engineers, and control theorists have also turned to evolutionary game theory, seeking tools for modeling dynamics in multiagent systems. Population Games and Evolutionary Dynamics provides a point of entry into the field for researchers and students in all of these disciplines. The text first considers population games, which provide a simple, powerful model for studying strategic interactions among large numbers of anonymous agents. It then studies the dynamics of behavior in these games. By introducing a general model of myopic strategy revision by individual agents, the text provides foundations for two distinct approaches to aggregate behavior dynamics: the deterministic approach, based on differential equations, and the stochastic approach, based on Markov processes. Key results on local stability, global convergence, stochastic stability, and nonconvergence are developed in detail. Ten substantial appendixes present the mathematical tools needed to work in evolutionary game theory, offering a practical introduction to the methods of dynamic modeling. Accompanying the text are more than 200 color illustrations of the mathematics and theoretical results; many were created using the Dynamo software suite, which is freely available on the author's Web site. Readers are encouraged to use Dynamo to run quick numerical experiments and to create publishable figures for their own research.

title: Evolutionary Dynamics: Exploring the Equations of Life
abstract: Preface 1.  Evolutionary Games 5. Prisoners of the Dilemma 6. Finite Populations 7. Games in Finite Populations 8.   The Evolution of Virulence 12. The Evolutionary Dynamics of Cancer 13.  Reading References Index

title: Cycles and Instability in a Rock-Paper-Scissors Population Game: a Continuous Time Experiment ∗
abstract: We report laboratory experiments that use new, visually oriented software to explore the dynamics of 3x3 games with intransitive best responses. Each moment, each player is matched against the entire population, here 8 human subjects. A "heat map" offers instantaneous feedback on current profit opportunities. In the continuous slow adjustment treatment, we see distinct cycles in the population mix. The cycle amplitude, frequency and direction are consistent with the standard learning models. Cycles are more erratic and higher frequency in the instantaneous adjustment treatment. Control treatments (using simultaneous matching in discrete time) replicate previous results that exhibit weak or no cycles. Average play is approximated fairly well by Nash equilibrium, and an alternative point prediction, "TASP" (Time Average of the Shapley Polygon), captures some regularities that Nash equilibrium misses.

title: The rock–paper–scissors game and the evolution of alternative male strategies
abstract: MANY species exhibit colour polymorphisms associated with alternative male reproductive strategies, including territorial males and 'sneaker males' that behave and look like females1–3. The prevalence of multiple morphs is a challenge to evolutionary theory because a single strategy should prevail unless morphs have exactly equal fitness4,5 or a fitness advantage when rare6,7. We report here the application of an evolutionary stable strategy model to a three-morph mating system in the side-blotched lizard. Using parameter estimates from field data, the model predicted oscillations in morph frequency, and the frequencies of the three male morphs were found to oscillate over a six-year period in the field. The fitnesses of each morph relative to other morphs were non-transitive in that each morph could invade another morph when rare, but was itself invadable by another morph when common. Concordance between frequency-dependent selection and the among-year changes in morph fitnesses suggest that male interactions drive a dynamic 'rock–paper–scissors' game7.

title: Red dominates black: agonistic signalling among head morphs in the colour polymorphic Gouldian finch
abstract: Recent sexual selection studies on the evolution of bird colouration have mainly focused on signals with a high level of condition-dependent variation, with much less attention given to colour traits whose expression is genetically controlled. Here, we experimentally tested the relative importance of a genetic colour polymorphism in determining male dominance in the Gouldian finch (Erythrura gouldiae), a species displaying three completely discrete but naturally co-occurring genetically inherited phenotypes; yellow-, red- (carotenoid) and black-headed (melanin) morphs. First, in staged dominance contests between unfamiliar birds of different head morphs, red-headed males dominated black-headed males, both of which dominated the yellow-headed birds. Second, within morphs, the intensity and size of the strongly ultraviolet-blue collar determined the outcome of these contests, and among the red-headed males, redder males dominated less chromatic birds. Lastly, when the dominance signal of red-headed birds was experimentally destabilized (i.e. blackened or reddened), naturally red-headed morphs continued to dominate both the black-and yellow-headed morphs. Together, these results suggest that intrinsic dominance-related behavioural differences between the three colour morphs, which are likely to influence the relative fitness of each morph, contribute to the complex selective patterns maintaining these three discrete phenotypes in relatively stable frequencies in wild populations.

title: Local dispersal promotes biodiversity in a real-life game of rock–paper–scissors
abstract: One of the central aims of ecology is to identify mechanisms that maintain biodiversity. Numerous theoretical models have shown that competing species can coexist if ecological processes such as dispersal, movement, and interaction occur over small spatial scales. In particular, this may be the case for non-transitive communities, that is, those without strict competitive hierarchies. The classic non-transitive system involves a community of three competing species satisfying a relationship similar to the children's game rock–paper–scissors, where rock crushes scissors, scissors cuts paper, and paper covers rock. Such relationships have been demonstrated in several natural systems. Some models predict that local interaction and dispersal are sufficient to ensure coexistence of all three species in such a community, whereas diversity is lost when ecological processes occur over larger scales. Here, we test these predictions empirically using a non-transitive model community containing three populations of Escherichia coli. We find that diversity is rapidly lost in our experimental community when dispersal and interaction occur over relatively large spatial scales, whereas all populations coexist when ecological processes are localized.

title: The Use of Knowledge in Society
abstract: AbstractWhat is the problem we wish to solve when we try to construct a rational economic order?On certain familiar assumptions the answer is simple enough. If we possess all the relevant information, if we can start out from a given system of preferences and if we command complete knowledge of available means, the problem which remains is purely one of logic. That is, the answer to the question of what is the best use of the available means is implicit in our assumptions. The conditions which the solution of this optimum problem must satisfy have been fully worked out and can be stated best in mathematical form: put at their briefest, they are that the marginal rates of substitution between any two commodities or factors must be the same in all their different uses.

title: Evolution and Game Theory
abstract: Research in noncooperative game theory has focused attention on two questions: Should we expect equilibrium play? If so, which of the multiple equilibria that arise in many games should we expect? This paper summarizes recent approaches to these questions that have been based on evolutionary models.

title: Equilibrium in Evolutionary Games: Some Experimental Results
abstract: Evolutionary game theory informs the design and analysis of twenty-six experimental sessions using normal form games with six to twenty-four players. The state typically converges to the subset of Nash equilibria called evolutionary equilibria, especially under conditions of mean matching and history. Mixed strategy equilibria are explained better by 'purification' strategies than by homogenous independent individual randomization. The risk dominance criterion fares poorly in some coordination game environments. With small player populations and large gains to cooperative behavior, some players apparently attempt to influence other players' actions, contrary to a key theoretical assumption.

title: What Does it Take to Eliminate the use of a Strategy Strictly Dominated by a Mixture?
abstract: This paper reports an experiment to determine whether subjects will learn to stop using a strictly dominated strategy that can be an above average reply. It is difficult to find an experimental design that eliminates the play of the strictly dominated strategy completely. The least effective treatment used money to motivate behavior directly. The most effective treatment used a binary-lottery with money prizes to induce preferences, but even this treatment required giving subjects plenty of experience. Doing so reduced the play of the strictly dominated strategy to around 10 percent by the end of a session. There is no evidence for the explosive cycling needed to make the strictly dominated strategy an above average reply.

title: A comparison of learning and replicator dynamics using experimental data
abstract: We compare the explanatory power of two popular adjustment models, replicator dynamics and simple belief learning dynamics. The data, gathered over 18 laboratory sessions each with 8 to 16 profit-motivated players, comprise more than 100 runs (each with 10 or more stages or periods) of single and two population 2×2 bimatrix games with unique interior equilibria. To equalize the number of free parameters we suppress individual player and treatment effects known to be highly significant in the simple belief learning model. Nevertheless the belief learning model narrowly outperforms the replicator model on the single population data and greatly outperforms it on the two population data.

title: Optimization Incentives and Coordination Failure in Laboratory Stag Hunt Games
abstract: This paper reports an experiment comparing three stag hunt games that have the same best-response correspondence and the same expected payoff from the mixed equilibrium, but differ in the incentive to play a best response rather than an inferior response. In each game, risk dominance conflicts with payoff dominance and selects an inefficient pure strategy equilibrium. We find statistically and economically significant evidence that the differences in the incentive to optimize help explain observed behavior.

title: Does Minimax Work? An Experimental Study
abstract: This paper finds experimental support for Von Neumann's minimax theory in two-person, zero-sum games.

title: Learning in games with unstable equilibria
abstract: We propose a new concept for the analysis of games, the TASP, which gives a precise prediction about non-equilibrium play in games whose Nash equilibria are mixed and are unstable under fictitious play-like learning. We show that, when players learn using weighted stochastic fictitious play and so place greater weight on recent experience, the time average of play often converges in these "unstable" games, even while mixed strategies and beliefs continue to cycle. This time average, the TASP, is related to the cycle identified by Shapley [L.S. Shapley, Some topics in two person games, in: M. Dresher, et al.

title: Near-equilibrium measurements of nonequilibrium free energy.
abstract: A central endeavor of thermodynamics is the measurement of free energy changes. Regrettably, although we can measure the free energy of a system in thermodynamic equilibrium, typically all we can say about the free energy of a nonequilibrium ensemble is that it is larger than that of the same system at equilibrium. Herein, we derive a formally exact expression for the probability distribution of a driven system, which involves path ensemble averages of the work over trajectories of the time-reversed system. From this we find a simple near-equilibrium approximation for the free energy in terms of an excess mean time-reversed work, which can be experimentally measured on real systems. With analysis and computer simulation, we demonstrate the accuracy of our approximations for several simple models.

title: Entropy production of cyclic population dynamics.
abstract: Entropy serves as a central observable in equilibrium thermodynamics. However, many biological and ecological systems operate far from thermal equilibrium. Here we show that entropy production can characterize the behavior of such nonequilibrium systems. To this end we calculate the entropy production for a population model that displays nonequilibrium behavior resulting from cyclic competition. At a critical point the dynamics exhibits a transition from large, limit-cycle-like oscillations to small, erratic oscillations. We show that the entropy production peaks very close to the critical point and tends to zero upon deviating from it. We further provide analytical methods for computing the entropy production which agree excellently with numerical simulations.

title: Symmetry properties of the large-deviation function of the velocity of a self-propelled polar particle.
abstract: A geometrically polar granular rod confined in 2D geometry, subjected to a sinusoidal vertical oscillation, undergoes noisy self-propulsion in a direction determined by its polarity. When surrounded by a medium of crystalline spherical beads, it displays substantial negative fluctuations in its velocity. We find that the large-deviation function (LDF) for the normalized velocity is strongly non-Gaussian with a kink at zero velocity, and that the antisymmetric part of the LDF is linear, resembling the fluctuation relation known for entropy production, even when the velocity distribution is clearly non-Gaussian. We extract an analogue of the phase-space contraction rate and find that it compares well with an independent estimate based on the persistence of forward and reverse velocities.

title: Stationary Concepts for Experimental 2 X 2 Games: Comment
abstract: Reinhard Selten and Thorsten Chmura (2008) recently reported laboratory results for completely mixed 2 X 2 games used to compare Nash equilibrium with four other stationary concepts: quantal response equilibrium, action-sampling equilibrium, payoff-sampling equilibrium, and impulse balance equilibrium. We reanalyze their data, correct some errors, and find that Nash clearly fits worst while the four other concepts perform about equally well. We also report new analysis of other previous experiments that illustrate the importance of the loss aversion hardwired into impulse balance equilibrium: when the other non-Nash concepts are augmented with loss aversion, they outperform impulse balance equilibrium.

title: Handbook of Experimental Economics Results
abstract: Markets market economics of uncertainty general equilibrium and the economics games mechanisms design and policy applications non-market and organizational research institutional choice and the evolution individual choice, beliefs and behaviour methods, classroom applications.

title: Volunteering leads to rock–paper–scissors dynamics in a public goods game
abstract: Collective efforts are a trademark of both insect and human societies. They are achieved through relatedness in the former and unknown mechanisms in the latter. The problem of achieving cooperation among non-kin has been described as the ‘tragedy of the commons’, prophesying the inescapable collapse of many human enterprises. In public goods experiments, initial cooperation usually drops quickly to almost zero. It can be maintained by the opportunity to punish defectors or the need to maintain good reputation. Both schemes require that defectors are identified. Theorists propose that a simple but effective mechanism operates under full anonymity. With optional participation in the public goods game, ‘loners’ (players who do not join the group), defectors and cooperators will coexist through rock–paper–scissors dynamics. Here we show experimentally that volunteering generates these dynamics in public goods games and that manipulating initial conditions can produce each predicted direction. If, by manipulating displayed decisions, it is pretended that defectors have the highest frequency, loners soon become most frequent, as do cooperators after loners and defectors after cooperators. On average, cooperation is perpetuated at a substantial level.

title: Chaos and unpredictability in evolutionary dynamics in discrete time.
abstract: A discrete-time version of the replicator equation for two-strategy games is studied. The stationary properties differ from those of continuous time for sufficiently large values of the parameters, where periodic and chaotic behavior replace the usual fixed-point population solutions. We observe the familiar period-doubling and chaotic-band-splitting attractor cascades of unimodal maps but in some cases more elaborate variations appear due to bimodality. Also unphysical stationary solutions can have unusual physical implications, such as the uncertainty of the final population caused by sensitivity to initial conditions and fractality of attractor preimage manifolds.

title: Intrinsic noise in game dynamical learning.
abstract: Demographic noise has profound effects on evolutionary and population dynamics, as well as on chemical reaction systems and models of epidemiology. Such noise is intrinsic and due to the discreteness of the dynamics in finite populations. We here show that similar noise-sustained trajectories arise in game dynamical learning, where the stochasticity has a different origin: agents sample a finite number of moves of their opponents in between adaptation events. The limit of infinite batches results in deterministic modified replicator equations, whereas finite sampling leads to a stochastic dynamics. The characteristics of these fluctuations can be computed analytically using methods from statistical physics, and such noise can affect the attractors significantly, leading to noise-sustained cycling or removing periodic orbits of the standard replicator dynamics.

title: Entropy production in nonequilibrium steady states: a different approach and an exactly solvable canonical model.
abstract: We discuss entropy production in nonequilibrium steady states by focusing on paths obtained by sampling at regular (small) intervals, instead of sampling on each change of the system's state. This allows us to directly study entropy production in systems with microscopic irreversibility. The two sampling methods are equivalent otherwise, and the fluctuation theorem also holds for the different paths. We focus on a fully irreversible three-state loop, as a canonical model of microscopic irreversibility, finding its entropy distribution, rate of entropy production, and large deviation function in closed analytical form, and showing that the observed kink in the large deviation function arises solely from microscopic irreversibility.

title: The Statistical Mechanics of Strategic Interaction
abstract: I study strategic interaction among players who live on a lattice. Each player interacts directly with only a finite set of neighbors, but any two players indirectly interact through a finite chain of direct interactions. I examine various stochastic strategy revision processes, including (myopic) best response and stochastic choice. I discuss both stationary distributions and the limit behavior of these Markov processes. Stationary distributions are partially characterized, and the asymptotic behavior of stochastic choice for those processes whose choice rule is nearly best-response is related to equilibrium selection in symmetric 2 × 2 and n × n coordination games.

title: Multiple equilibria and limit cycles in evolutionary games with Logit Dynamics
abstract: This note shows, by means of two simple, three-strategy games, the existence of stable periodic orbits and of multiple, interior steady states in a smooth version of the Best-Response Dynamics, the Logit Dynamics. The main finding is that, unlike Replicator Dynamics, generic Hopf bifurcation and thus, stable limit cycles, occur under the Logit Dynamics, even for three-strategy games. We also show that the Logit Dynamics displays another bifurcation which cannot occur under the Replicator Dynamics: the fold bifurcation, with non-monotonic creation and disappearance of steady states.

title: Quantal Response Equilibria for Normal Form Games
abstract: We investigate the use of standard statistical models for quantal choice in a game theoretic setting. Players choose strategies based on relative expected utility and assume other players do so as well. We define a quantal response equilibrium (ORE) as a fixed point of this process and establish existence. For a logit specification of the error structure, we show that as the error goes to zero, QRE approaches a subset of Nash equilibria and also implies a unique selection from the set of Nash equilibria in generic games. We fit the model to a variety of experimental data sets by using maximum likelihood estimation.

title: Hysteresis effects of changing parameters of noncooperative games
abstract: We adapt the method used by Jaynes to derive the equilibria of statistical physics to instead derive equilibria of bounded rational game theory. We analyze the dependence of these equilibria on the parameters of the underlying game, focusing on hysteresis effects. In particular, we show that by gradually imposing individual-specific tax rates on the players of the game, and then gradually removing those taxes, the players move from a poor equilibrium to one that is better for all of them.

title: Evolutionary game dynamics
abstract: Evolutionary game dynamics is the application of population dynamical methods to game theory. It has been introduced by evolutionary biologists, anticipated in part by classical game theorists. In this survey, we present an overview of the many brands of deterministic dynamical systems motivated by evolutionary game theory, including ordinary differential equations (and, in particular, the replicator equation), differential inclusions (the best response dynamics), difference equations (as, for instance, fictitious play) and reaction-diffusion systems. A recurrent theme (the so-called `folk theorem of evolutionary game theory') is the close connection of the dynamical approach with the Nash equilibrium, but we show that a static, equilibrium-based viewpoint is, on principle, unable to always account for the long-term behaviour of players adjusting their behaviour to maximise their payoff.

title: Human strategy updating in evolutionary games
abstract: Evolutionary game dynamics describe not only frequency-dependent genetic evolution, but also cultural evolution in humans. In this context, successful strategies spread by imitation. It has been shown that the details of strategy update rules can have a crucial impact on evolutionary dynamics in theoretical models and, for example, can significantly alter the level of cooperation in social dilemmas. What kind of strategy update rules can describe imitation dynamics in humans? Here, we present a way to measure such strategy update rules in a behavioral experiment. We use a setting in which individuals are virtually arranged on a spatial lattice. This produces a large number of different strategic situations from which we can assess strategy updating. Most importantly, spontaneous strategy changes corresponding to mutations or exploration behavior are more frequent than assumed in many models. Our experimental approach to measure properties of the update mechanisms used in theoretical models will be useful for mathematical models of cultural evolution.

title: Dynamics of Boltzmann Q learning in two-player two-action games.
abstract: We consider the dynamics of Q learning in two-player two-action games with a Boltzmann exploration mechanism. For any nonzero exploration rate the dynamics is dissipative, which guarantees that agent strategies converge to rest points that are generally different from the game's Nash equlibria (NEs). We provide a comprehensive characterization of the rest point structure for different games and examine the sensitivity of this structure with respect to the noise due to exploration. Our results indicate that for a class of games with multiple NEs the asymptotic behavior of learning dynamics can undergo drastic changes at critical exploration rates. Furthermore, we demonstrate that, for certain games with a single NE, it is possible to have additional rest points (not corresponding to any NE) that persist for a finite range of the exploration rates and disappear when the exploration rates of both players tend to zero.

title: Probability of second law violations in shearing steady states.
abstract: We propose a new definition of natural invariant measure for trajectory segments of finite duration for a many-particle system. On this basis we give an expression for the probability of fluctuations in the shear stress of a fluid in a nonequilibrium steady state far from equilibrium. In particular we obtain a formula for the ratio that, for a finite time, the shear stress reverse sign, violating the second law of thermodynamics. Computer simulations support this formula.