# The motor Wisdom of the Crowd

Gabriel Madirolas, Regina Zaghi-Lara, Alex Gomez-Marin, Alfonso Pérez-Escudero

<figure><img src="image_1.png" /></figure>

## To cite this version:

Gabriel Madirolas, Regina Zaghi-Lara, Alex Gomez-Marin, Alfonso Pérez-Escudero. The motor Wisdom of the Crowd. Journal of the Royal Society Interface, 2022, 19 (195), 10.1098/rsif.2022.0480. hal-03856360

HAL Id: hal-03856360

https://hal.science/hal-03856360

Submitted on 20 Nov 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
INTERFACE

royalsocietypublishing.org/journal/rsif

Research

<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /></figure>

Check for
updates

**Cite this article:** Madirolas G, Zaghi-Lara R, Gomez-Marin A, Pérez-Escudero A. 2022
The motor Wisdom of the Crowd. *J. R. Soc. Interface* 19: 20220480.
https://doi.org/10.1098/rsif.2022.0480

Received: 29 June 2022
Accepted: 15 September 2022

**Subject Category:**
Life Sciences–Mathematics interface

**Subject Areas:**
computational biology, biomathematics,
biocomplexity

**Keywords:**
Wisdom of the Crowd, collective intelligence,
trajectory aggregation, visuomotor control,
touchscreen tracing

**Authors for correspondence:**
Alex Gomez-Marin
e-mail: agomezmarin@gmail.com
Alfonso Pérez-Escudero
e-mail: alfonso.perez-escudero@univ-tlse3.fr

Electronic supplementary material is available online at https://doi.org/10.6084/m9 figshare. c.6214763.

The motor Wisdom of the Crowd

Gabriel Madirolas<fnref n="1" />,†, Regina Zaghi-Lara<fnref n="2" />,†, Alex Gomez-Marin<fnref n="2" />,‡,<fnref n="3" /> and Alfonso Pérez-Escudero<fnref n="1" />,‡

<figure><img src="image_4.png" /></figure>

GM, 0000-0002-4526-4507; AG-M, 0000-0003-2764-2583; AP-E, 0000-0002-4782-6139

Wisdom of the Crowd is the aggregation of many individual estimates to obtain a better collective one. Because of its enormous social potential, this effect has been thoroughly investigated, but predominantly on tasks that involve rational thinking (such as estimating a number). Here we tested this effect in the context of drawing geometrical shapes, which still enacts cognitive processes but mainly involves visuomotor control. We asked more than 700 school students to trace five patterns shown on a touchscreen and then aggregated their individual trajectories to improve the match with the original pattern. Our results show the characteristics of the strongest examples of Wisdom of the Crowd. First, the aggregate trajectory can be up to 5 times more accurate than the individual ones. Second, this great improvement requires aggregating trajectories from different individuals (rather than trials from the same individual). Third, the aggregate trajectory outperforms more than 99% of individual trajectories. Fourth, while older individuals outperform younger ones, a crowd of young individuals outperforms the average older one. These results demonstrate for the first time Wisdom of the Crowd in the realm of motor control, opening the door to further studies of human and also animal behavioural trajectories and their mechanistic underpinnings.

1. Introduction

The Wisdom of the Crowd (WOC) is the notion that the aggregate opinion of a diverse group of people may be more reliable than that of an expert. This idea gained great standing in the academic community in 1906 when Sir Francis Galton showed that the average of hundreds of individual estimates about the weight of an ox matched its actual weight within 1%—far more accurately than a highly skilled farmer [1]. The potential of this effect was rapidly appreciated, but applications were limited for a long time, due to the practical difficulty of gathering and aggregating individual opinions. The rapid development of communication technologies since the 1990s has removed that barrier, unleashing the potential of the WOC, which is now replacing individual experts in several realms, from collectively written Wikipedia articles that replace classical encyclopaedia articles written by experts [2] to Stack Exchange posts written and voted by users that replace traditional manuals [3]. These examples show the transformative potential of the WOC.

However, to properly grasp the non-trivial scope of the WOC and to ensure its validity, it is critical to understand the effect deeply. In particular, two key questions must be answered. The first key question is how to extract a WOC estimate from a group of people. In the traditional paradigm, subjects were as diverse as possible [4], each subject made an independent estimate (they did not communicate with each other) [5], and opinions were aggregated by averaging all individual guesses [6]. The condition of independence is traditionally regarded as crucial to guarantee that systematic individual errors cancel out

© 2022 The Authors. Published by the Royal Society under the terms of the Creative Commons Attribution License http://creativecommons.org/licenses/by/4.0/, which permits unrestricted use, provided the original author and source are credited.

<footnote>†These authors contributed equally to this work.</footnote>

<footnote>‡These authors jointly supervised the work.</footnote>

<footnote n="1">¹Research Centre on Animal Cognition (CRCA), Centre for Integrative Biology (CBI), Toulouse University, CNRS, UPS, 31062 Toulouse, France</footnote>

<footnote n="2">²Behavior of Organisms Laboratory, Instituto de Neurociencias de Alicante (CSIC-UMH), Alicante, Spain</footnote>

<footnote n="3">³The Pari Center, via Tozzi 7, 58045 Pari (GR), Italy</footnote>
when aggregating several individuals [5,6]. An example of
the negative effect of broken independence arises when sub-
jects are informed of the guesses of others and are then
allowed to emit their guess or reconsider their previous one
[7]. However, quantifying social influence over each individ-
ual allows one to find new aggregation measures that
counteract this effect [8], and to take advantage of it to
improve upon the crowd estimate [9]. Moreover, it has been
shown how the condition of independence can be relaxed
[10], and how allowing subjects to discuss before arriving at
a consensus estimate can lead to improvements at both the
group and individual level [11,12]. Others have argued for
maximal differences (negative correlations) between subjects
as the essential requisite for the WOC [13], and therefore
the detection of correlations is presented as a powerful tool
to improve the collective estimate when it deviates from
the true value [14]. Finally, methods to find subgroups of
individuals whose aggregated estimate may be better than
the aggregated estimate of the whole crowd have been
proposed, for example, based on identifying expertise
within the crowd [15].

The second key question is what tasks can be performed
more efficiently by a collective than by an individual. Classical
demonstrations of the WOC consisted of guessing a number or
choosing over a set of discrete alternatives [5,6]. While a big
part of research still follows this paradigm, many studies
have successfully applied WOC to more complex tasks, such
as estimation of multi-dimensional quantities [16], sequential
decision-making [17], collective guessing of a sentence [18],
collective editing [2], forecasting in prediction markets and
prediction polls [19,20], correct tempo of a classical music
piece [21], medical diagnosis [22,23] or drug prescription
[24]. However, the vast majority of these studies share a
common characteristic: they focus on explicitly rational tasks,
in which individuals need to make a conscious estimate.

Here we investigated whether WOC can be applied to a
task that depends on embodied motor control rather than
high-order abstract cognition. To that end, we developed an
experimental paradigm where we asked children to trace a
series of predefined patterns on a tablet. Such a situation
defies classical WOC studies because it is a complex motor
task, which is difficult to parameterize or describe in simple
terms, and whose errors are highly correlated (a deviation
at any point in the line affects the future trajectory of the
finger). It is also worth emphasizing that the task is inti-
mately related to drawing, an important part of human
culture and, with the appropriate experimental design and
measuring tools, can be tested in naturalistic conditions
beyond artificial laboratory settings. Although some studies
have investigated collective problem-solving in tasks invol-
ving movement [25], visual decision-making [26-28] and
visual search [29-31], our study is, to the best of our knowl-
edge, the first one showing the implications of aggregating
individual solutions to a sensory-motor task.

In this paper, we first present experimental results from
hundreds of subjects tracing with their fingers a series of
well-defined geometrical templates displayed on touchscreen
tablets in a classroom setting. Using such 'big behavioural
data' [32] collected 'outside the laboratory' [33], we examine
the four main features that characterize the strong instances
of WOC. The paper is thus organized along these four ques-
tions: (i) whether individual trajectories of subjects can be
aggregated to produce a more accurate description of the

desired pattern, (ii) whether the improvement is a true
'crowd' effect (requiring different individuals, as opposed
to a single individual repeating the same task), (iii) whether
the effect is strong enough so that the aggregate is more accu-
rate than most of the individuals and (iv) whether the effect is
strong enough so that a crowd of low-skill individuals out-
performs one high-skill individual. We find that all these
conditions are met, providing the first evidence of motor
WOC.

2. Results

## 2.1. Collecting behavioural big data in a classroom setting

We asked 797 school students with ages between 6 and 18
years old to trace with their finger several shapes using a
custom-made drawing app (figure 1a). After a few minutes
of practice to familiarize themselves with the app, subjects
were invited to reproduce five different geometric curves
with varying levels of complexity, from ellipses to four-fold
rose figures. A template of each shape was shown in the
screen of the tablet (figure 1a), and subjects were instructed
to trace fluidly and continuously for 30 s, not excessively
fast so as to avoid systematically overshooting the template
but not excessively slow so as to avoid halts and brief jerky
movements in trying to perfectly match the template. In
other words, to simply produce good-enough tracing. All
shapes were closed curves that could be traced repeatedly
in a single stroke, and almost every subject traced each tem-
plate several times during the 30 s of each experimental
curve (figure 1b; electronic supplementary material, figure
S1a). Our experimental procedure allowed us to test hun-
dreds of children producing thousands of high-resolution
trajectories (more than 24 h of data) in naturalistic conditions
(see Methods for more details).

## 2.2. Processing the trajectories to extract the Wisdom of the Crowd

In order to estimate the WOC of the trajectories traced by our
subjects, the first step consisted in splicing the full trajectory
of each subject into individual trajectories that represent a
single pass over the template. This task turned out to be
complicated for some complex shapes, and especially for
low-accuracy trajectories that deviated very much from the
template. To avoid biasing our results and increase the trans-
parency of our analysis, we resorted to a simple approximate
method: we determined that in most cases subjects traced
each template at least 8 times, and divided the full
trajectory of each subject into 8 segments of equal duration,
which we then call 'raw individual trajectories' (electronic
supplementary material, figure S1b). While in most cases
each raw individual trajectory contains more than a single
pass over the template, this fact does not affect our
conclusions and, in fact, strengthens them (see Methods).

Next, we subsampled the templates and processed the
raw individual trajectories to facilitate their further analysis.
We first took a set of reference points along each template
(figure 1c, left; electronic supplementary material, figure
S1c). We then took the raw individual trajectory, assigned
each of its points to the nearest reference point on the
<figure><img src="image_5.png" /><figcaption>**Figure 1.** Description of the experiment and computation of errors. (a) One of the participants tracing on a tablet. The template was always present in thick black line on blue background. Participants were asked to trace the template with the index finger of their dominant hand continuously for 30 s. The procedure took place simultaneously in groups of 30 students in a classroom setting. (b) Example of template (black dashed line), and trajectories from three different subjects (blue, red and green solid lines). (c) Method to compute errors and aggregate trajectories. Black lines represent the reference trajectory, which is divided in segments of equal length (limited by dashed lines). Each black dot is the median centre of mass of the reference trajectory for each segment. Blue points: experimental individual trajectories. The individual error is computed as the distance between the points for the individual trajectory and the respective centres of mass for the reference trajectory. Red points are the median centre of mass of several individual trajectories (blue points) within the same segment. The Wisdom of the Crowd trajectory is formed with these averages (red line), and the error for the Wisdom of the Crowd is computed as the distance between these averages and the corresponding centres of mass of the reference trajectory. See electronic supplementary material, figure S1, for a detailed description of trajectory subsampling, aggregation of individual trajectories and computation of trajectory errors.</figcaption></figure>

template, and found the median centre of mass of each group of points (figure 1c, left; electronic supplementary material, figure S1d-h). In this way, we obtained a subsampled individual trajectory (which we simply call ‘individual trajectory’), with one experimental point corresponding to each reference point of the template. From now on, we will refer to these subsampled individual trajectories simply as ‘individual trajectories’.

To compute WOC trajectories, we computed the median centre of mass for the points associated with the same reference point from different individual trajectories (figure 1c, right; electronic supplementary material, figure S1i-k).

To quantify the accuracy of a trajectory (either an individual trajectory or a WOC trajectory), we computed the distance between each of its points and the corresponding reference point of the template (figure 1c; electronic supplementary material, figure S1l), obtaining the distance between them for each small region. Then, we used the average of these errors to quantify the overall error for the whole trajectory (electronic supplementary material, figure S1m).

We are now in a position to establish whether the drawing task fulfils the four criteria of WOC mentioned above.

## 2.3. Criterion (i): Wisdom of the Crowd trajectories are more accurate than individual trajectories

We first investigated the accuracy of individual trajectories. Participants cared about properly tracing the templates but were not particularly motivated to be accurate, as we instructed them to trace quickly and fluidly without being too concerned about accuracy. Furthermore, our dataset includes data from very young children to late teenagers, whose motor skills are at different maturation stages. Consequently, individual trajectories showed a lot of dispersion (figure 2a, blue).

In spite of this inaccuracy at the individual level, WOC achieves remarkable accuracy. We built WOC trajectories for all shapes by taking one individual trajectory from each subject and aggregating all of them (figure 2a, red). Average error (across all patterns) decreased more than twofold, from
<figure><img src="image_6.png" /><figcaption>**Figure 2.** WOC in pattern tracing. (a) Original template (black dashed line), individual trajectories (blue), and WOC trajectory resulting from the aggregation of all individual trajectories (red). (b) Average error of individual trajectories for each region of the templates. (c) Error of the WOC trajectories for each region of the templates. (d) Improvement of the WOC over the individual trajectories for each region of the templates (i.e. average error of individual trajectories minus error of the WOC trajectory).</figcaption></figure>

2.72 mm for individual trajectories (figure 2b) to 1.03 mm for WOC trajectories (figure 2c). Improvement of WOC over individual trajectories was unequal over different portions of the templates (figure 2d). Despite not showing a definite structure, one can notice interesting differences in the local improvement along different parts of the templates. In ellipses, major improvements take place in the straightest parts of the curve, where tracing speed is typically faster. While three-petal flowers show their WOC maximal improvement at the edges, four-petal flowers have it where curvature is minimal (rather than at the inner high-curvature turns). In both cases, however, the improvement corresponds to the most distal parts of the template taken globally. In the lemniscate, the WOC shows an intriguing global top-bottom asymmetry.

## 2.4. Criterion (ii): Wisdom of the Crowd accuracy improves with a diverse crowd

Aggregating several trajectories from a single subject should also lead to an improvement in accuracy, an effect termed ‘the crowd within’ [34]. Therefore, the improvement reported here for the WOC trajectories might not require a diverse crowd, but just be a consequence of aggregating multiple datasets (regardless of whether they come from the same subject or from different ones).

To elucidate whether our observation is a true effect of the crowd, we took advantage of the fact that each subject traced each pattern several times. While in the previous section we built our WOC trajectory from all of our subjects, here we studied how the error of the WOC trajectory changes as a function of how many individual trajectories are aggregated. We also compared the case of aggregating trajectories from the same individual (cycles while the same person draws the same pattern) or from different individuals. While in both cases the error decreases as we aggregate more

trajectories, this decrease is much more rapid when the trajectories come from different individuals (figure 3a), thus better supporting the WOC in movement trajectories.

This advantage of crowds over repeated trials of the same subject indicates that subjects tend to repeat their own errors. Different trajectories drawn by the same subject tend to deviate in the same regions and towards the same side, creating systematic biases. These biases are corrected when aggregating trajectories from different subjects, whose deviations are more balanced.

We also used this procedure to determine how many subjects are needed to achieve high accuracy. The error decreases monotonically as we add more trajectories from different subjects, dropping by almost half with only 10 subjects and starting to saturate after 50 subjects (figure 3a, red).

## 2.5. Criterion (iii): Wisdom of the Crowd outperforms most individuals

A mere improvement of accuracy as we aggregate more trajectories would seem insufficient to support our claim of a WOC effect in motor control. In fact, it is a mathematical necessity that the error of an aggregate trajectory must be equal to or smaller than the average error of the individual trajectories [14,35].

What makes WOC such an important effect is the magnitude of the improvement. While there is no absolute threshold to consider that the effect is strong enough, a classical criterion is that the WOC estimate must be better than an overwhelming majority of the individual estimates. This was the case for example in Galton’s original demonstration of the effect, in which the arithmetic mean of individual guesses was better than every single individual estimate [36].

To investigate whether our WOC trajectories outperform the vast majority of individual estimates, we computed all
<figure><img src="image_7.png" /><figcaption>**Figure 3.** A strong case of WOC. Each column corresponds to one template, in the same order as shown in figure 2. (a) Average error of the aggregate trajectory, as a function of the number of individual trajectories that are aggregated. Black: aggregated trajectories belong to the same subject. Red: aggregated trajectories come from different subjects. Pale patches represent the 95% confidence interval (calculated via bootstrapping). (b) Histogram of errors for individual trajectories, compared with the error of the WOC trajectory (red line). (c) Average individual error for children and teens (blue), and error of the WOC trajectory for children and teens (red). Error bars represent the 95% confidence interval (calculated via bootstrapping).</figcaption></figure>

individual errors for each template and compared it with the error of the WOC estimate. In all cases, the WOC estimate outperforms 99.5% of the individual trajectories, except in one pattern in which it outperforms 95% of the subjects (figure 3b). Therefore, our dataset meets the criterion that the WOC estimate outperforms the vast majority of the individual estimates.

## 2.6. Criterion (iv): Wisdom of the Crowd of low-skill individuals beats the average high-skill individual

The last criterion of strong WOC, which is critical for its practical applicability, is that a crowd of low-skill individuals must outperform a single high-skill individual. To test this criterion, we first need to divide our population of subjects in groups with different expected skill levels. For tasks that require specific skills learned through education or professional training, high-skill subjects can be defined through their cultural level or profession. In the context of drawing, we considered testing professional painters and designers, but these professions usually have more to do with an aesthetic sense and the ability to master different drawing tools than with the motor control required to follow a predefined line with one's finger.

However, our empirical approach allowed us to sample a large population of maturing subjects of different ages. Such a rich dataset offered a practical criterion to separate subjects by skill level: age stratification. Our results include subjects aged from 6 to 18 years (figure 4a), and motor control develops during this period (especially between 6 and 10 years of age) [37–40]. Indeed, our results show that individual performance improves with age (figure 4b). Therefore, we defined low-skill individuals as young children (less than 10.5 years), and high-skill individuals as older children (greater than 10.5 years) whose motor skills are comparably more developed (while this threshold maximizes the difference in individual performance between the two groups, our results hold regardless of the threshold chosen; electronic supplementary material, figure S2).

As a confirmation of this criterion, individual estimates are better for older children than for younger ones (figure 3c, compare bars 1 and 2 of every template). The key question is then whether a crowd of young children outperforms the average older child. Remarkably, we found this to be true for every template (figure 3c, compare bars 2 and 3 of every template). Therefore, our dataset meets the last criterion of the strongest versions of WOC: a crowd of low-skill individuals outperforms the average high-skill individual.
<figure><img src="image_8.png" /><figcaption>**Figure 4.** Effect of age. (a) Number of subjects of each age in our dataset. (b) Average error of the individual trajectories (averaged over all segments of each template, over individuals with same age, and over the 5 templates) as a function of age. Error bars represent the 95% confidence interval (calculated via bootstrapping).</figcaption></figure>

## 2.7. High-skill individuals play little role in the motor
Wisdom of the Crowd

We also found an interesting and unexpected result. When comparing the results from WOC estimates from old and young children, we found almost no difference. Only for 2 of the 5 templates are the WOC estimates better for older children than for younger ones (figure 3c, compare bars 3 and 4 of every template). This result suggests that, when recruiting a crowd to perform a WOC estimate, there is no benefit in selecting high-skill individuals [41]. In other words, one may not need to include (motor) ‘experts’ to achieve the (motor) WOC.

We also asked the question of whether different criteria to define high-skill or low-skill individuals could give different results. To answer this question, we simulated criteria that estimated individual skill with different accuracies, from those capable of selecting the very best individual, to benchmarks incapable of distinguishing skill. Our results turned out to be very robust: for a high-skill individual to outperform the crowd, the selection method must be able to identify accurately the top 1.73% performers (electronic supplementary material, figure S3).

## 2.8. The method to aggregate trajectories has little
impact on the results

An important practical question is how to aggregate the individual responses to produce the WOC estimate [7,14], and in particular whether to use the mean or the median,

considering that the latter can counteract the effect of outliers [42,43]. All our previous analyses use the median to compute centres of mass, as we detected some subjects with very large deviations from the templates. We re-did our analysis using the mean instead of the median and compared both methods, to find very little difference between them (electronic supplementary material, figure S4). This result indicates that, even though some subjects are clear outliers, the distribution of trajectories around the true pattern has relatively little skew (figure 2a; note how the density of traces is nearly symmetric along the templates).

## 3. Discussion

Here we tested whether WOC can take place in a context far removed from those studied so far. By asking our subjects to trace a complex trajectory rather than estimating a number, we investigated a procedure that does not consist in making an explicit rational estimate but in performing an implicit embodied motor task. We tested hundreds of children drawing in a custom-made tablet app, collecting a large amount of precise and quantitative data to test the effect of aggregating independent individuals for better collective performance.

We found that tracing geometrical patterns manifests all the characteristics of WOC: (i) accuracy improves when aggregating several trajectories, (ii) these trajectories must come from different subjects, (iii) the aggregate trajectory outperforms most individual ones and (iv) a crowd of low-skill individuals outperforms a typical high-skill individual. Our results thus extend the concept of WOC from its classical application of quantity estimation to the realm of motor control and embodied cognition.

Our findings suggest that WOC may be applicable with a greater generality than previously thought because they address outstanding theoretical concerns. For the WOC estimate to be accurate, the individual estimates must follow a probability distribution whose average (either the mean or the median) matches the true value. A general concern was that the tasks typically chosen, such as number estimation, might share some characteristics that made them fulfil this requirement, which would not be met when trying to extend WOC to other contexts. Here we have tested WOC in a completely different context, in which the object to be estimated is not a number but a complex trajectory, and the estimation procedure is not a rational thought process but a motor task. Our results indicate that the conditions required for WOC are met with great generality, and call for a more systematic exploration across tasks of different nature.

In our results, collectives of younger children achieve almost the same accuracy as collectives of older ones, despite the fact that the former are on average less accurate individually. This result indicates that individual inaccuracy does not translate here to a larger bias at the population level: younger children draw more diverse trajectories (average individual Euclidean distance to collective trajectory 2.77 mm for younger, 2.37 mm for older; see electronic supplementary material, figure S5), but their aggregate is as accurate as that of older children. Since more diverse datasets converge to their average faster than less diverse ones (by virtue of the diversity prediction theorem [5]), collectives of younger children catch up with the accuracy of collectives of older
ones. The fact that young children exhibit no larger systematic bias than older ones suggests that young children might have a similar psychomotor plan to execute the task than older ones, but it is executed with less precision due to underdeveloped motor control.

Different contexts may modulate the effect. Our results were collected after instructing subjects to emphasize natural movements rather than accuracy. Different instructions might change the results, as greater emphasis on individual accuracy might decrease the impact of WOC, while absence of instructions might increase variability (and therefore the potential for WOC effects). In any case, we found a high accuracy in the motor WOC, despite the complex nature of the task, the extreme inaccuracy of some trajectories and the fact that errors accumulate (a deviation at one point in the trajectory affects its future). This high accuracy opens the door to applying similar aggregation methods to other tasks with similar characteristics, such as finding the optimal trajectory of a vehicle [44] or in the context of skill improvement in amateur and professional athletes, where the aggregation of different patterns used to execute a movement or a manoeuvre could complement other approaches aiming to find an optimal biomechanical technique [45].

Our results suggest a potential application in the study of human symbols. Letters, numbers and other symbols are written differently by different people [46], and their shape changes significantly over time and from region to region [47]. These changes are well documented qualitatively, and recent techniques have been developed to characterize them quantitatively [48,49]. These techniques are usually based on aggregating images, regardless of the trajectories that the writers followed to trace each symbol. Our results indicate that it would be possible to compute an average trajectory for a given symbol across a population. The methodology would need to be different from the one presented in this paper, with trajectory estimation and alignment posing important methodological challenges. But the surprisingly high accuracy found in our dataset indicates that aggregating symbols written by different people may provide an intelligible average tracing trajectory, which would facilitate the study of how handwriting changes in space and time.

# 4. Methods

## 4.1. Hardware

Thirty Android touch-screen tablets were used for the behavioural experiments. The tablet brand was Samsung Galaxy Tab A6 (size: 254 × 164 × 8 mm³, Android v. 8.1.0 and API level 26). The price per tablet was less than $200. The display has a 10.1-inch PLS LCD screen, with dimensions 216 × 135 mm², and a resolution of 1920 × 1200 px². The tablet has a capacitive touch-screen, and registers touch by a finger or a capacitive stylus, with a resolution equal to the display resolution. Maximum screen refresh rate is 60 Hz, and maximum sampling rate of touch events is close to 85 Hz.

## 4.2. Software

The app was programmed in Android Studio (v. 3.3.2) in the Kotlin programming language [33], and tailored specifically for accuracy, efficiency and robustness in out-of-the-laboratory experiments with children. It can be freely downloaded

((https://github.com/adam-matic/KinematicCognition), and also edited for different experimental purposes.

## 4.3. Experimental procedures

A total of 851 subjects (56% female, 10% left-handed) participated in the experiment, most of whom were school students between 6 and 18 years old (797 subjects; see electronic supplementary material, table S1). All experiments were performed during the 2019 Brain Awareness Week (from 11 to 15 March 2019). Students arrived in groups of about 30 individuals, belonging to the same school class. Classes belonged to several different schools in the area of Alicante, Spain. There were no specific selection criteria for schools and classes beyond their willingness to participate in our experiment and a more or less homogeneous sampling of ages and locations. Groups were assigned different time slots throughout the morning. The experiments took place in a regular small classroom (with a capacity for 30 students) in a building adjacent to the Instituto de Neurociencias de Alicante, Spain. We used 30 tablets (a single tablet per children), placed on the tables in the classroom. Each experimental session had an overall duration of about 15 min.

Students were invited to enter the classroom and sit down wherever they wished. Then, before starting the experiments, students were greeted and briefly told about the overall goal of the study (they already knew some details because they had an explanation of the activity days before by their teachers at their own local schools). The explanation was generic, namely, a brief and fun experiment was going to take place where they would simply need to draw and trace specific geometric figures in order as they would appear on the tablet screen, helping scientists to study motor control (this fitted well in the Brain Awareness Week, since each group also received a related outreach talk, so that they could not only listen about scientific experiments but actually also participate in them).

Students were requested to use the index finger of their dominant hand to draw and trace on the tablet screen (rather than using tablet pens). They were also asked to avoid touching the screen with the other hand, or to move the tablet from where it was placed when they entered the room. Before starting the experiment, they all tapped on the screen at the custom-made app logo of the experiment.

First, a simple screen opened where they were asked whether their dominant hand was left or right, their age by scrolling on the date of birth, and their gender. After the information is complete, the app allows two options: 'Practice' or 'Experiment'.

Second, the participants did a trial exercise before the actual experiment, where similar curves appeared to the ones they would encounter later. This part was key for them to familiarize with the tasks they would need to accomplish next. In particular, they were instructed not to separate the finger from the screen until each task was over, and to perform fluid movements, avoiding delineating too slow or too fast. They could at this point ask questions before the experiment took place.

Third, after such trials, oral instruction prompted the students to start all the 'Experiment' part at the same time. The experimental part consisted of a series of exercises, or tasks, all automatically concatenated with brief pauses in between. In this way, we avoided having to verbally interrupt the whole classroom with various unnecessary (and potentially distracting and confusing) instructions (especially for the younger children) to start the many different drawing and tracing tasks.

Although sitting next to each other, participants had little influence on each other's performance. While the attitude of other participants might affect general features such as the speed or care with which they performed the task, the nature of our experiment made it virtually impossible to use information from other subjects to increase one's performance.
Three different classes of tasks were presented to the partici-
pants: tracing, tracking and scribbling. Every class comprised
different exercises, each one with a duration of 30 s, with a 7 s
pause in between. Thus, the whole motor control experience
was brief, avoiding distractions or loss of interest by the children.
A visual summary of the experimental dataset can be found at:
https://youtu.be/rz-TWk_6HSU.

In this study, we only analysed the first class of task (tracing),
where participants had to delineate with their finger a black
curve on a blue screen. The curves shown were an oval (or
ellipse), a thinner oval (larger eccentricity), three-petal and four
petal-flowers (both based on Huh's pure frequency curves
[50,51]), and an infinite symbol (lemniscate). Subjects were not
instructed to start tracing at any specific location of the templates.
However, the duration of each drawing trial ensured that all
participants covered the same surface area, several times.

When the experiment finished, students were thanked again
and invited to leave the room to continue enjoying the Brain
Awareness Week parade at the Instituto de Neurociencias. Exper-
imental procedures were approved by the Institutional Review
Board and followed the required guidelines on participation
and personal data protection. Parents of the students had been
previously informed and asked for written consent to record
the finger trajectories.

4.4. Data cleaning

We removed anomalous data following three criteria. (i) Trajec-
tories that lasted less than 25 s, which indicated subjects who
were not focused on the task, since they were instructed to
draw for the full duration of the experiment (less than 4% of tra-
jectories were removed for this reason, and most of them were
shorter than 15 s, so including them would have minimal
impact in our conclusions). (ii) Trajectories whose standard devi-
ation (in either the horizontal or vertical dimension) was less
than half the standard deviation of the points of the template.
(iii) Trajectories with jumps between two consecutively recorded
points greater than 1/4 of the dimensions of the touchscreen,
which in most cases were due to malfunctioning of the tablet
or the subject touching the screen with both hands simul-
taneously. See electronic supplementary material, table S1, for
the number of full trajectories originally stored, the number of
trajectories that did not meet each of the filtering criteria (some
trajectories did not meet more than one criterion), and the
number of trajectories that were finally used for the analysis. In
sum, a total of 3485 trajectories were analysed, each with a dur-
ation longer than 25 s, which yields an estimate of a total of more
than 24 h of high-resolution quantitative measurements of
human drawing in naturalistic conditions.

4.5. Definition of raw individual trajectories

Each subject traced the template repeatedly during the 30 s allotted for the task, so each original trajectory contains several passes over the template. The ideal definition for ‘individual trajectory’ would be a single pass over the template, but finding the exact point in which the trajectory finishes one pass and starts the next is problematic: when the trajectory is a very poor approximation to the template, it is unclear when we should consider that an individual trajectory has ended. Therefore, attempts to divide the trajectories in this way would either force us to discard the less accurate trajectories or could result in systematic biases affecting differently the high- and low-accuracy trajectories.

For this reason, we chose a simple definition of raw individ-
ual trajectory: we divided each trajectory in 8 segments of equal
duration, and we took each of these segments as a single raw
individual trajectory. While this is only an approximation, it
has the advantage of being a simple and transparent method
and avoiding biases among trajectories of different accuracy.

Because of this approximate definition, in many cases, an
individual trajectory contains only part of the template or more
than one pass over some parts of the template. To determine a
convenient number of segments, we randomly extracted 150
original trajectories (from only those where a single pass could
be unambiguously defined). In 76% of the cases, subjects com-
pleted 8 or more full passes over the whole template (and only
6% of subjects performed 4 or 3 passes) so on average our indi-
vidual trajectories contain more than one pass. We made this
choice to be conservative: since they in fact contain more than
one pass over each template (with only 17% including 2 or
more passes), our individual trajectories already benefit from
a small degree of aggregation, and therefore their error is
smaller than the one we would measure if we took a true
single pass. Therefore, the effect of WOC in our dataset is,
if anything, probably underestimated. See electronic supplemen-
tary material, figures S6 and S7, for a demonstration of how the
results are similar when dividing the original trajectories in 5 or
12 segments.

4.6. Subsampling of individual trajectories

Raw individual trajectories contained between 200 and 215
points. Before any analysis, we subsampled the templates into
a set of reference points. We used 50 reference points for the
two ellipses and 100 reference points for the rest; these numbers
were chosen to be high enough to represent each template faith-
fully, while low enough to ensure that sufficient experimental
points fell into each bin. Then, we subsampled the raw individ-
ual trajectories to ensure that each point of a subsampled
individual trajectory corresponded to one reference point of its
corresponding template. We did this by finding the median
centre of mass of all points of the raw individual trajectory near-
est to each reference point of the template (similar results are
obtained by using the arithmetic mean instead of the median,
as shown in electronic supplementary material, figure S4). See
electronic supplementary material, figure S1, for a step-by-step
description of this process. The term ‘individual trajectory’ here-
inafter refers to the subsampled one (the term ‘raw individual
trajectory’ referred to the trajectory before subsampling).

4.7. Aggregation of individual trajectories

To aggregate several individual trajectories (regardless of
whether they belonged to the same subject or to different sub-
jects), we computed the median centre of mass of the points of
each individual trajectory corresponding to the same reference
point in the template (figure 1c). The final result is an aggregate
trajectory that has the same number of points as the number of
reference points of the template (see electronic supplementary
material, figure S1).

4.8. Computation of errors

To compute the error of any trajectory (either an individual tra-
jectory or an aggregated one), we found the Euclidean distance
between each of its points and the corresponding reference
point of the template (figure 1c; electronic supplementary
material, figure S1). These distances are used to represent the col-
ours in figure 2b-d, showing the average error for each region of
each template across all the trajectories. The total error of a trajec-
tory is computed as the arithmetic mean of all the distances to
the reference points.

To compute the average error when aggregating individual
trajectories from the same subject (figure 3a, black lines), we
created all possible sets with a given number of individual trajec-
tories from each subject. For example, when aggregating 3
individual trajectories, for each subject there are 56 different
combinations out of the total of 8 individual trajectories, so
there are 56 different sets of 3 individual trajectories per subject. The error for the template is determined as the average of the errors over all sets of each subject, and then over all subjects.

To compute the average error when aggregating individual trajectories from different subjects (as for example in figure 3a, red lines), it would not be possible to compute all possible combinations (there would be too many). Therefore, we performed a random sample: we randomly drew the desired number of subjects from the database, and for each subject, we randomly chose one of the individual trajectories. Then, we aggregated the individual trajectories, and computed the error of the aggregate one. We repeated this process 10 000 times and computed the arithmetic mean of all errors.

To compute the average error across templates in figure 4, we first computed the total error for each of the 5 templates and then computed the arithmetic mean of these 5 error values.

## 4.9. Computation of confidence intervals

To compute the confidence intervals for the errors of individual and aggregated trajectories (figure 3a,c), we used bootstrapping [52]. First, we created ‘virtual experiments’ by randomly drawing subjects with repetition until we reached the total number of subjects. Therefore, each virtual experiment consisted of the same number of subjects, but due to the random sampling some of our original subjects may be missing, and some subjects may be present more than once. Then, we recreated all our analysis on each of these virtual experiments. We repeated this full process with 500 virtual experiments, and our confidence intervals represent the region that contains 95% of these results.

**Ethics.** Experimental procedures were approved by the Institutional Review Board of the Miguel Hernández University in Elche (Alcantara, Spain), under project registration number 2019.111.E.OIR, and followed the required guidelines on participation and personal data protection. Parents of the students had been previously

informed and asked for written consent to record their children's finger trajectories.

**Data accessibility.** Experimental data and code for analysis and figure generation can be found at the following link: https://github.com/gabrielmadirolas/motorwoc.

The data are provided in electronic supplementary material [53].
**Authors' contributions.** G.M.: conceptualization, formal analysis, investigation, methodology, software, visualization, writing—original draft, writing—review and editing; R.Z.-L.: conceptualization, data curation, investigation, methodology, validation, writing—review and editing; A.G.-M.: conceptualization, data curation, funding acquisition, investigation, methodology, project administration, resources, software, supervision, validation, writing—review and editing; A.P.-E.: conceptualization, formal analysis, funding acquisition, investigation, methodology, project administration, software, supervision, validation, writing—original draft.

All authors gave final approval for publication and agreed to be held accountable for the work performed therein.

**Conflict of interest declaration.** We declare we have no competing interests.

**Funding.** This work was funded by the Spanish Ministry of Science (grants BFU2015-74241-JIN and RYC-2017-23599 to A.G.-M.; pre-doctoral contract from RYC-2017-23599 funds to R.Z.-L.), by the Severo Ochoa Center of Excellence programmes (grant no. SEV-2013-0317 start-up funds to A.G.-M.), by a CNRS-Momentum grant to A.P.-E., by the Gore Family Foundation through a start-up grant to A.P.-E. and by the Fyssen Foundation through a Research grant to A.P.-E.

**Acknowledgements.** We want to especially thank Adam Matic who, jointly with A.G.-M., designed the tablet app for big behavioural data collection in the real world [33]. He programmed the tablet app and participated in the design of the experiments, and processed the trajectory data. We thank María del Carmen Lillo Navarro for assistance during some of the experiments. We are also grateful to Alicia Ferri and Diego Echevarría for the general organization of the Brain Awareness Week at the Instituto de Neurociencias de Alicante. We thank members of IVEP at the Research Center for Animal Cognition in Toulouse, France, for feedback on a previous version of the manuscript.

## References

1. Galton F. 1907 Vox populi. *Nature* **75**, 450–451. (doi:10.1038/075450a0)

2. Kittur A, Chi E, Pendleton BA, Suh B, Mytkowicz T. 2007 Power of the few vs. wisdom of the crowd: Wikipedia and the rise of the bourgeoisie. *World Wide Web* **1**, 19.

3. Burghardt K, Alsina EF, Girvan M, Rand W, Lerman K. 2017 The myopia of crowds: cognitive load and collective evaluation of answers on Stack Exchange. *PLoS ONE* **12**, e0173610. (doi:10.1371/journal.pone.0173610)

4. Page SE. 2008 *The difference: how the power of diversity creates better groups, firms, schools, and societies*. Princeton, NJ: Princeton University Press.

5. Hong L, Page SE. 2012 Some microfoundations of collective wisdom. In *Collective wisdom: principles and mechanisms* (eds H Landemore, J Elster), pp. 56–71. Cambridge, UK: Cambridge University Press.

6. Surowiecki J. 2005 *The wisdom of crowds*. New York, NY: Anchor.

7. Lorenz J, Rauhut H, Schweitzer F, Helbing D. 2011 How social influence can undermine the wisdom of crowd effect. *Proc. Natl Acad. Sci. USA* **108**, 9020–9025. (doi:10.1073/pnas.1008636108)

8. Kao AB, Berdahl AM, Hartnett AT, Lutz MJ, Bak-Coleman JB, Ioannou CC, Giam X, Couzin ID. 2018 Counteracting estimation bias and social influence to improve the wisdom of crowds. *J. R. Soc. Interface* **15**, 20180130. (doi:10.1098/rsif.2018.0130)

9. Madirolas G, de Polavieja GG. 2015 Improving collective estimations using resistance to social influence. *PLoS Comput. Biol.* **11**, e1004594. (doi:10.1371/journal.pcbi.1004594)

10. Miller B, Steyvers M. 2011 The wisdom of crowds with communication. In *Proc. Annual Meeting of the Cognitive Science Society, Boston, MA, USA, 20–23 July 2011*.

11. Ioannou CC, Madirolas G, Brammer FS, Rapley HA, de Polavieja GG. 2018 Adolescents show collective intelligence which can be driven by a geometric mean rule of thumb. *PLoS ONE* **13**, e0204462. (doi:10.1371/journal.pone.0204462)

12. Navajas J, Niella T, Garbulsky G, Bahrami B, Sigman M. 2018 Aggregated knowledge from a small number of debates outperforms the wisdom of large crowds. *Nat. Hum. Behav.* **2**, 126–132. (doi:10.1038/s41562-017-0273-4)

13. Davis-Stober CP, Budescu DV, Dana J, Broomell SB. 2014 When is a crowd wise? *Decision* **1**, 79. (doi:10.1037/dec0000004)

14. Laan A, Madirolas G, De Polavieja GG. 2017 Rescuing collective wisdom when the average group opinion is wrong. *Front. Robot. AI* **4**, 56. (doi:10.3389/frobt.2017.00056)

15. Budescu DV, Chen E. 2014 Identifying expertise to extract the wisdom of crowds. *Manag. Sci.* **61**, 267–280. (doi:10.1287/mnsc.2014.1909)

16. Yi SKM, Steyvers M, Lee MD, Dry MJ. 2012 The wisdom of the crowd in combinatorial problems. *Cogn. Sci.* **36**, 452–470. (doi:10.1111/j.1551-6709.2011.01223.x)

17. Thomas B, Coon J, Westfall HA, Lee MD. 2021 Model-based wisdom of the crowd for sequential decision-making tasks. *Cogn. Sci.* **45**, e13011. (doi:10.1111/cogs.13011)

18. Clément RJ, Krause S, Von Engelhardt N, Faria JJ, Krause J, Kurvers RHJM. 2013 Collective cognition in humans: groups outperform their best members in a sentence reconstruction task. *PLoS ONE* **8**, e77943. (doi:10.1371/journal.pone.0077943)
19. Ray R. 2006 Prediction markets and the financial 'Wisdom of Crowds'. *J. Behav. Finance* **7**, 2–4. (doi:10.1207/s15427579jpfm0701_1)

20. Atanasov P, Rescober P, Stone E, Swift SA, Servan-Schreiber E, Tetlock P, Ungar L, Mellers B. 2017 Distilling the wisdom of crowds: prediction markets vs. prediction polls. *Manag. Sci.* **63**, 691–706. (doi:10.1287/mnsc. 2015.2374)

21. Martin-Castro A, Ucar I. 2020 Conductors' tempo choices shed light over Beethoven's metronome. *PLoS ONE* **15**, e0243616. (doi:10.1371/journal.pone. 0243616)

22. Winkler RL, Poses RM. 1993 Evaluating and combining physicians' probabilities of survival in an intensive care unit. *Manag. Sci.* **39**, 1526–1543. (doi:10.1287/mnsc.39.12.1526)

23. Kattan MW, O'Rourke C, Yu C, Chagin K. 2016 The wisdom of crowds of doctors: their average predictions outperform their individual ones. *Med. Decis. Making* **36**, 536–540. (doi:10.1177/ 0272989X15581615)

24. Krockow EM, Kurvers RH, Herzog SM, Kämmer JE, Hamilton RA, Thilly N, Macheda G, Pulcini C. 2020 Harnessing the wisdom of crowds can improve guideline compliance of antibiotic prescribers and support antimicrobial stewardship. *Sci. Rep.* **10**, 18782. (doi:10.1038/s41598-020- 75063-z)

25. Moussaid M et al. 2012 Traffic instabilities in self-organized pedestrian crowds. *PLoS Comput. Biol.* **8**, e1002442. (doi:10.1371/journal.pcbi. 1002442)

26. Bahrami B, Olsen K, Latham PE, Roepstorff A, Rees G, Frith CD. 2010 Optimally interacting minds. *Science* **329**, 1081–1085. (doi:10.1126/science. 1185718)

27. Mahmoodi A et al. 2015 Equality bias impairs collective decision-making across cultures. *Proc. Natl Acad. Sci. USA* **112**, 3835–3840. (doi:10.1073/pnas. 1421692112)

28. Bang D et al. 2017 Confidence matching in group decision-making. *Nat. Hum. Behav.* **1**, 0117. (doi:10. 1038/s41562-017-0117)

29. Juni MZ, Eckstein MP. 2015 Flexible human collective wisdom. *J. Exp. Psychol. Hum. Percept. Perform.* **41**, 1588. (doi:10.1037/xhp0000101)

30. Juni MZ, Eckstein MP. 2017 The wisdom of crowds for visual search. *Proc. Natl Acad. Sci. USA* **114**, E4306–E4315. (doi:10.1073/pnas.1700010114)

31. Saha Roy T, Mazumder S, Das K. 2021 Wisdom of crowds benefits perceptual decision making across difficulty levels. *Sci. Rep.* **11**, 538. (doi:10.1038/ s41598-020-80500-0)

32. Gomez-Marin A, Paton JJ, Kampff AR, Costa RM, Mainen ZF. 2014 Big behavioral data: psychology, ethology and the foundations of neuroscience. *Nat. Neurosci.* **17**, 1455–1462. (doi:10.1038/nn.3812)

33. Matic A, Gomez-Marin A. 2019 A customizable tablet app for hand movement research outside the lab. *J. Neurosci. Methods* **328**, 108398. (doi:10. 1016/j.jneumeth.2019.108398)

34. Vul E, Pashler H. 2008 Measuring the crowd within: probabilistic representations within individuals. *Psychol. Sci.* **19**, 645–647. (doi:10.1111/j.1467- 9280.2008.02136.x)

35. Larrick RP, Soll JB. 2006 Intuitions about combining opinions: misappreciation of the averaging principle. *Manag. Sci.* **52**, 111–127. (doi:10.1287/mnsc.1050. 0459)

36. Wallis KF. 2014 Revisiting Francis Galton's forecasting competition. *Stat. Sci.* **29**, 420–424. (doi:10.1214/14-STS468)

37. Blank R, Miller V, Von Voss H. 2000 Human motor development and hand laterality: a kinematic analysis of drawing movements. *Neurosci. Lett.* **295**, 89–92. (doi:10.1016/S0304-3940(00)01592-5)

38. Bryden PJ, Roy EA. 2005 Unimodal performance across the age span. *Brain Cogn.* **57**, 26–29. (doi:10. 1016/j.bandc.2004.08.016)

39. Visser J, Geuze RH, Kalverboer AF. 1998 The relationship between physical growth, the level of activity and the development of motor skills in adolescence: differences between children with DCD and controls. *Hum. Mov. Sci.* **17**, 573–608. (doi:10. 1016/S0167-9457(98)00014-1)

40. Zeng N, Ayyub M, Sun H, Wen X, Xiang P, Gao Z. 2017 Effects of physical activity on motor skills and

cognitive development in early childhood: a systematic review. *BioMed Res. Int.* **2017**, 2760716. (doi:10.1155/2017/2760716)

41. Hong L, Page SE. 2004 Groups of diverse problem solvers can outperform groups of high-ability problem solvers. *Proc. Natl Acad. Sci. USA* **101**, 16 385–16 389. (doi:10.1073/pnas.0403723101)

42. Hooker RH. 1907 Mean or median. *Nature* **75**, 487–488. (doi:10.1038/075487a0)

43. Galton F. 1907 The ballot-box. *Nature* **75**, 509–510. (doi:10.1038/075509f0)

44. Melo J, Naftel A, Bernardino A, Santos-Victor J. 2006 Detection and classification of highway lanes using vehicle motion trajectories. *IEEE Trans. Intell. Transp. Syst.* **7**, 188–200. (doi:10.1109/TITS.2006.874706)

45. Glazier PS, Mehdizadeh S. 2019 In search of sports biomechanics' holy grail: can athlete-specific optimum sports techniques be identified? *J. Biomech.* **94**, 1–4. (doi:10.1016/j.jbiomech.2019.07.044)

46. Srihari SN, Cha S-H, Arora H, Lee S. 2002 Individuality of handwriting. *J. Forensic Sci.* **47**, 1–17. (doi:10.1520/JFS15447J)

47. Yeandle L. 1980 The evolution of handwriting in the English-speaking colonies of America. *Am. Arch.* **43**, 294–311.

48. Stokes PA. 2015 Digital approaches to paleography and book history: some challenges, present and future. *Front. Digit. Humanit.* **2**, 5. (doi:10.3389/ fdigh.2015.00005)

49. Ciula A. 2005 Digital palaeography: using the digital representation of medieval script to support palaeographic analysis. *Digit. Mediev.* **1**, 1.

50. Huh D. 2015 The vector space of convex curves: how to mix shapes. (https://arxiv.org/abs/1506.07515)

51. Huh D, Sejnowski TJ. 2015 Spectrum of power laws for curved hand movements. *Proc. Natl Acad. Sci. USA* **112**, E3950–E3958. (doi:10.1073/pnas. 1510208112)

52. Efron B, Tibshirani RJ. 1993 *An introduction to the bootstrap*. New York, NY: Chapman & Hall.

53. Madirolas G, Zaghi-Lara R, Gomez-Marin A, Pérez- Escudero A. 2022 Data from: The motor Wisdom of the Crowd. Figshare. (doi:10.6084/m9.figshare.c. 6214763)