Theory-testing experiments in the economics laboratory

Abstract: Features of experimental design impose auxiliary hypotheses onexperimenters. Hertwig & Ortmann rightly argue that the ways some vari-ables are implemented in psychology cloud results, whereas the differentimplementations in economics provide for more robust results. However ,not all design variables support this general conclusion. The repetition oftrials may confuse results depending on what theory is being tested. W eexplore this in the case of simple bargaining games. The conclusions that we draw from theory-testing experiments aretempered by what auxiliary hypotheses we have about the prob-lem domain that are relevant to the experiment. If a theory pre-dicts that P, and in our experiment we observe not-P , before weconclude that the theory does not square with the facts we mustbe able to rule out that it is some auxiliary hypothesis H (perhapsa hypothesis about how our experiment is run) tacitly appealed tothat is confounding the prediction of the theory (Duhem 1906;Quine 1953b). Hertwig & Ortmann (HO repe-tition of trials may confuse rather than clarify experimental resultsdepending on the target theory .Experimental practices in economics and psychology differ and,in particular , some of the differences can be described as varyingthe extent to which procedures serve as confounding hypothesesin theory-testing. If we are interested in rational choice (both inpure decision problems and strategic environments), why shouldwe think that what subjects say they would do should track whatthey actually would do if given financial incentives? After all, thatis what makes choices interesting in real life: something turns onwhat it is we choose. The lack of financial incentives clouds ex-perimental results; an experiment in which subjects are paid a flatfee imposes a rather implausible auxiliary hypothesis into thestory . Similar problems arise for some of the other design variablesthat H&O discuss, like the use of deception by experimenters andproviding a well-defined decision environment (what they call aOscriptO). Deceiving subjects, for instance, imposes the implausi-ble auxiliary hypotheses that experimenters have no reputationamong subjects and that subjects will complete a given task in anexperiment even if they are suspicious about what the OrealO taskof the experiment is. The basic conclusion for experiments with-out a clearly defined script is roughly the same.The same general sort of conclusion cannot be drawn with re-spect to whether an experiment allows for repeated trials or not,as H&O mistakenly think. W e are most interested in the case ofexperimental games. The authors argue that Oequilibrium is as-sumed not to be reached right awayO but that Oit is expected toevolveO over time until a steady-state in observed behavior isreached. And this is meant to be the reason why economists insiston the use of repeated trials. This is erroneous for at least two rea-sons.First, this argument gets the history of game theory backwards.The traditional concept of equilibria is that they represent whereideally rational agents would play , given certain information con-ditions. These agents, being ideally rational, can be expected toreach an optimal outcome immediately . Equilibria, in this classi-cal interpretation, are supposed to characterize the set of possibleoutcomes that such agents could reach by deliberating about thestructure of the game. On the other hand, the OlearningO inter-pretation of equilibria in games is quite recent, and representsOthe alternative explanation that equilibrium arises as the long-runoutcome of a process in which less than fully rational players gropefor optimality over timeO (Fudenberg & Levine 1998, p. 1). So itis a mistake to think that the traditional interest in equilibria re-quires that experiments use repeated trials.Second, depending on what theory a given experiment is meantto be testing, repeated trials may impose rather than eliminate animplausible auxiliary hypothesis into the story . T ake the case ofsimple two-person trust games (Berg et al. 1995). H&O suggestthat the Oattention-grabbing results of these games [might be] dueto the very fact that they are typically implemented as one-shotrather than repeated games.O They offer a distinction between twosorts of repeated environments, stationary replications and whatmay be called repetition with replacement. But if we take the sug-gestion that our experimental design imposes auxiliary hypothesesinto the mix seriously , we have to consider whether either of thesesorts of repeated trials is appropriate to the theory being tested.In the case of bargaining experiments, where the understood the-ory being tested is the standard noncooperative solution conceptof Nash equilibrium and its refinements, neither of these types ofrepeated trials is essential.Consider first, repetition with replacement. In these games,subjects may meet one another more than once. The future mayloom large enough that cooperative play is supported by the FolkTheorem. But then, observing cooperative behavior in this envi-ronment would not distinguish between noncooperative gametheory and more exotic solution concepts (e.g., trust and reci-procity). So if we want to test the noncooperative story , this envi-ronment is not best suited to that purpose.