Data Gathering to Build and Validate Small-Scale Social Models for Simulation

ly studying decentralisation There is a contemporary general issue in economics: to construct a theory of behaviour for economic actions (Kirman, 1997). Researchers belonging to the trend called ACE (Agent-Based Computational Economics) have used multi-agent simulations (or “agent-based simulations”) either to test models that they had inferred from the observation of economic activities or, using genetic algorithm, to discover some optimal algorithms. The idea is to describe the rationality of agents in a really decentralised setting, with imperfect information, and look at the evolution of the global data so that to judge the relevance of the model (Tesfatsion, 2006). Up until recently, those focusing on the behaviour of individuals and history of transactions rather than optimality were quite rare, and their hypotheses were based on long-term experience in micro-economics (Kirman and Vriend, 2001). Other economists focus on a more precise description of rationality: for now more than 40 years, experimentalists have been testing the possibility to implement archetypical economic settings in laboratories (Smith, 2002). They make human subjects play in a game, where choices faced are similar to those described by economic theory. Those subjects’ preferences are voluntarily controlled by monetary incentives (and their rational action is computable by the one who organises the experiment); the amount of information they get is limited and carefully defined. The issue is to judge if subjects do behave in a perfectly rational way, or if they show some behavioural features that are not predicted by theory. The experiment has one advantage which is to be perfectly controlled and reproducible. All information circulation and actions are strictly limited and recorded with the help of computers. The situation is hence different from real life, where many situations are different from ideally expected equilibrium, but it is impossible to decide if the lack of information or the lack of rationality is responsible. The experimenter does not have access to the calculus led by the subject, but can make subtle changes in distributed information, to understand better the way it is used. The results have been gathered for some time and a huge amount of relevant descriptions on how humans treat information in economic contexts has been gathered (Camerer, 2003), concerning for example: cognitive limitations, need for reciprocity and altruist behaviours, consumer attitude, public good games, non-cooperative games. Most of the time, results are compared to theoretical expectations, and when real behaviour differs from optimal model, interpretation has to be given. To explain the deviations from theoretical behaviours, positive models are proposed, as dynamical learning in context based on equations (Fudenberg and Levine, 1999). These models are now tested thanks to simulations. Designing these simulations is pretty easy because the settings for experiments were already ready to be transposed to produce a distributed model. All proposed equations and hypothesis have to be turned into algorithms, and when the simulation has run, quantitative data can serve as benchmarks, and using the same indicators, help selecting models (Brenner and Vriend, 2005). The main part of produced research is dealing with learning in markets (Duffy, 2001; Rouchier and Robin, 2005; Brewer et al., 2002), but some were concerned with other topics, like cooperation (Janssen and Ahn, 2003). For an excellent review on the topic, one can refer to John Duffy’s chapter in the forthcoming handbook for computational economics (Duffy, 2006). Examples and critics An application of the abstract model referred to as EWA (Experience Weighted Attraction learning model) by Ho and Camerer, can be found in Chong et al (2005). The paper is a study of the calibration of the model with some data collected through experiments. The model is considered as very good and pure, since it only depends on one variable to adjust. Authors show that it is rather good for estimating learning by individuals, and that they can characterise different type of players (sophisticated – who realize that others are learning or not) by looking at their results. However, one can note that in the paper, they only consider rather simple games with two agents. Plus, the fit of simulated and experiments data is not so good. Some people tried to use their model in simulations using a higher number of agents, and found it rather impossible to choose the adequate parameters to fit their own data. This corresponds pretty well to the analysis by Thomas Brenner, who tries to make a complete review of the relevant use of learning models depending on the type of model that is built and shows that for the moment, each have very limited application (Brenner, 2006). Going further, Janssen and Ahn (Janssen and Ahn, 2003) showed that, in the context of a public good provision game, it is not possible to adapt existing individual learning models to both individual and global data. Using a large set of experimental results from diverse researchers with varying pay-offs in the same game, they ran simulations using two types of models: best-response model with signalling and EWA. Both models could match human results much better (statistically) than a perfect rationality calculus. EWA was proven to be good to match individual trajectories, but was very poor to find global results; best-response was good for the global results but could not describe individual trajectories well. A recent paper (Lelec, 2005) presents reimplementation of a set of simulations based on experiments on ultimatum game, with a reinforcement learning model proposed by Roth and Erev (1995). The paper shows that reinforcement learning is extremely sensitive to parameters settings. A very annoying result is that different parameters settings can lead to simulation results that are difficult to discriminate. Hence which model is right is difficult to choose just comparing with human subject experiments. The indicators that are usual for experimenters do not always allow discriminating, even for rather simple systems. It can be found also in the case of the reproduction of a market with artificial agents, where the speed of convergence is similar for a lot of different simulations and does not allow to decide which model is best (Rouchier and Robin, 2005). John Duffy (2001), in a complete research, reveals all these issues and looks for experiments to go further in understanding human rationality in context. His example was an attempt to lead human subjects, and then artificial agents, to speculate. In his first experiments, he recorded behaviours, asked his subjects about their logic and produced hypothesis on learning. From these assumptions he built an algorithm and led simulations. He insists on saying that, to compare, he used the same number of agents and the same time-length as in real experiments (which is rarely done). Since he could not match data for individual agents’ trajectories and global results, he mixed agents and subjects. Then all results, including human behaviours, changed a lot. In no case could he get his artificial agents to act like humans globally, only individual behaviours would qualitatively match humans’ ones. The need for new protocols The comparison of experiments’ results and simulation data seems quite easy to realise at first sight, due to the perfect reproducibility of experiments and the possibility to produce simulation settings where the representation of information circulation is very close to the one of experiments. The idea of building a general algorithm that mimics human learning is apparently not a reasonable short term aim in the contemporary context and might never be. When it comes to understanding better the structures that emerge through iterative learning, it seems clear that several issues are at stake. There is a need to rely on a strict methodology to build indicators and link results to each initial parameters of the simulation. To know the context of learning should enable to decide which learning model is better in different cases (Brenner, 2006). One can note that for the moment, representation of intelligence for humans is indeed very little inspired by some of the great theories of mind that have inspired the apparition of Artificial Intelligence as a field. Maybe the fact that all reasoning is represented through linear equation based models is a problem. The composition of mind as an emergence of different interacting sub-computers is a hypothesis that maybe should not be ignored when representing learning, even in economics (Minsky, 1985; Bateson, 1972). When writing about the importance of experiments, Vernon Smith (2002) stresses the fact that no experiment can actually destroy or prove a theory, just ask new questions, and identify application boundaries for theories. This is a rather usual assertion nowadays, and corresponds roughly to what is referred to as the Duhem-Quine thesis: experiments embed so many hypothesis that one result can not be taken into account to refute or assess a specific aspect. Simulations can be seen as having a similar status, and are just a way to test the coherence of hypothesis (or cognitive consistence) and show their limits, from which new assumptions should be expressed. The question of “truth” of a complex model seems so impossible to evaluate that many researchers now consider that it should not be used on its own, as an abstract object that can more or less prove anything, but only considered in its context of use (Barreteau et al., 2004). This is similar to the efficiency paradigm of the instrumentalist approach, also said to be emphasised by Quine (Zammito, ). A rather different protocol for research with simulations and human subjects has evolved in that sense, usually referred to as “companion modelling”, enables to accumulate knowledge on huma