South-East Asia (Graham et al., 2017). Thus, one might expect similar patterns to characterize micro-work whereby in the case of France, supply might largely come from French-speaking African countries following historical "coloniality" patterns (Casilli, 2017).

More importantly perhaps, the weakening of geographical boundaries exposes online micro-workers to broad competition, possibly producing over-supply of labor as has already been documented in the case of freelancing (Graham and Anwar, 2019). Wide swings in demand, dependent on the rapidly changing needs of the global artificial intelligence industry, exacerbate this tendency (Schmidt, 2019). This likely results in a very skewed distribution of levels of engagement, with many registered micro-workers whose activity is actually very low because they do not manage to align with task availability fluctuations.

Overall, data publicly provided by platforms, notably the number of their registered users, are more promising than general population surveys, but cannot be taken at face value. Corrections and adjustments are in order, and their interpretation calls for further thought.

## 2.3 The limitations of alternative approaches to estimating micro-work

To move beyond the limitations of data publicly provided by platforms, some researchers endeavored to interrogate them privately – assuming that platform owners have gut sense of the market in which they operate. Following this approach, Information Systems researchers at the University of Kassel looked at the case of Germany. They first mapped all platforms established in Germany and then surveyed them about their views of themselves and the whole sector. Notice that they did not focus on micro-workers but on “crowdworkers”, a notion that as noted above (footnote 1), encompasses both micro-workers and some types of freelancers. One question invited respondents to estimate the gross number of German crowdworkers throughout platforms. The average of all answers gives a rough figure of over one million, of which one quarter are allegedly “active”, though this notion is not precisely defined (Mrass and Peters, 2017). While interesting, this result is based on a low number of responses (17) with high variance.

Other studies explicitly probe publicly-announced company data, specifically the 500,000 announced by Amazon Mechanical Turk, the oldest and by far the best known platform. A key motivation of many of these studies is to assure the quality of the results of scientific surveys, questionnaires and experiments for which research teams in disciplines such as psychology, marketing, and linguistics recruit “Turkers” (Fort et al., 2011). This involves ensuring access to a sufficiently large pool of respondents (Keith et al., 2017). In this perspective, Stewart et al. (2015) borrowed the bio-ecological “capture-recapture” model (see section 3.3) to show the existence of a pool of 7,300 potential study participants on Mechanical Turk at any given time. In the same year, activist Kristy Milland conducted a six-week sur-
vey and counted about 30,000 workers<fnref n="8" />. Difallah et al. (2018) improved Stewart’s model, allowing for a longer observation time and taking into account the heterogeneous propensity of individuals to accept tasks. They showed that about 100,000 people work on Jeff Bezos’ platform, or one fifth of the advertised population. Finally, using data from TurkPrimer, an independent company that helps researchers design and implement studies on Mechanical Turk, a recent study finds 250,000 Turkers worldwide (Robinson et al., 2019).

Notice the shift of focus in these studies, which do not target a country across platforms, but a platform across countries. However, about 75% of Amazon Mechanical Turk workers are Americans (Difallah et al., 2018), and some of these estimates paid particular attention to them, partly to serve the needs of scientific studies that targeted US participants. Therefore, these findings say little on other platforms and on other countries – especially as Mechanical Turk is so unique by fame and longevity, that its characteristics may not transfer easily to other settings.

The immediate lesson to be learned from this brief review is that no method is perfect, but a combination of tools may ensure that they compensate for each other’s limitations. Leaving aside general population surveys due to their high cost and uncertain results, in what follows we start by mapping the platforms that operate in one specific country, France (section 3.1). Then (section 3), we retrieve the numbers that these platforms provide publicly, and endeavor to correct them to consider different levels of workers’ engagement as well as multi-activity. This will require deployment of a wider range of methods, including some of those previously discussed.

### 3 Three approaches to estimating the number of micro-workers in a specific country

In light of the above discussions, we now deploy a range of approaches to estimate the size of the French micro-working population. We start by identifying the micro-tasking platforms that recruit French workers (subsection 3.1). Then we follow the literature in referring to the figures publicly provided by these platforms (subsection 3.2), and then applying a “capture-recapture” model to a single platform (subsection 3.3). We then present an alternative method based on website audience tracking (subsection 3.4). We combine our different methods to improve accuracy and obtain a meaningful range of estimates (subsection 3.5) and finally, we refine our results by taking into account micro-workers performing tasks on several platforms (subsection 3.6).

<footnote n="8">⁸ The study is not published but the author has made her calculations available: Kristy Milland, 150717 Preliminary results - Mapping study, July 17, 2015, URL: https://docs.google.com/spreadsheets/d/1T3yP\_Jo4qELrwsE2NAPNs07L1AWmpAEr9vnhreGJ\-K0/edit\#gid=1993074859 (accessed 18 December 2018).</footnote>
## 3.1 What micro-work platforms operate in France?

To map the micro-work market in France, we cannot limit our investigation to Amazon Mechanical Turk, where demand from French companies outweighs the supply of French micro-workers. The platform has long discouraged registrations outside the USA and, to a lesser extent, India. Recent research (Berg et al., 2018; Forde et al., 2017) covering multiple countries including France, has added to Amazon Mechanical Turk also Crowdflower (see footnote 7), Microworkers, Clickworker<fnref n="9" /> and, in the case of Berg et al. (2018), Prolific. We include their selection of platforms except Prolific, a site specialized in surveys that exists only in English, and which does not seem to have users in France (see Berg et al. (2018)). We also retain ClixSense, which provides access to Crowdflower.

In addition to these international platforms, we include FouleFactory, a platform that exclusively recruits its micro-workers in France and since its creation in 2014, has established itself as a key player in the market. It sells services to artificial intelligence and other hi-tech companies through its sister start-up Wirk, launched in 2018. We also include Ferception, a French platform that recruits internationally. FouleFactory and Ferception are potentially more attractive for French speakers than other platforms like Microworkers, where English language is required even to sign up. Finally, we add Appen, an international platform open to recruitment in France, which lists tasks of the same nature as the others, but has a tighter and more closed business model that enables it to manage large orders for clients, and to offer more regular assignments to workers. As noted earlier (footnote 7), Appen owns the former Crowdflower but has left it to operate independently throughout 2019.

This list is not exhaustive and there are other micro-work websites and applications in France. We exclude micro-working platforms that seem to be very rarely used in the country, such as CrowdGuru and Spare5, and French platforms that are in-between micro-tasking and freelancing, such as Malt and 5euros.com, as the part of micro-work in their overall offer is small. We also exclude income-generating mobile applications such as BeMyEye and Roamler, which do offer micro-tasks, but of a particular nature that requires physical presence in a given place (for example, taking pictures of products in supermarkets), so that they are not online-only. Finally, we must unfortunately exclude platforms that are in scope, but for which we could not find sufficient data, notably Lionbridge and Pactera.

It is useful to characterize this selection of platforms according to their access policies (open or limited) and the presence of at least one welcome page in the local language (Table 1). With regard to the first criterion, notice that FouleFactory has closed registration of new members but reopens it intermittently in case of surge in the demand of micro-work by client firms. As for the second criterion, the presence of a French page seems to encourage access by new users.

<footnote n="9">⁹ In addition to connecting its own registered micro-workers and clients, Clickworker also provides an entry point for UHRS (Universal Human Relevance System), Microsoft’s proprietary micro-work platform, where it operates as a monopsony.</footnote>
<table><caption>**Table 1** Characteristics of micro-work platforms included in our study</caption><thead><tr><th>Platform</th><th>Open sign-up</th><th>Local language page</th></tr></thead><tbody><tr><td>Amazon Mechanical Turk</td><td>No</td><td>No</td></tr><tr><td>Microworkers</td><td>Yes</td><td>No</td></tr><tr><td>Clickworker</td><td>Yes</td><td>Yes</td></tr><tr><td>ClixSense</td><td>Yes</td><td>No</td></tr><tr><td>FouleFactory</td><td>No</td><td>Yes</td></tr><tr><td>Ferpection</td><td>Yes</td><td>Yes</td></tr><tr><td>Appen</td><td>No</td><td>Yes</td></tr></tbody></table>

*Source:* authors’ elaboration based on platforms’ websites.

## 3.2 Analyzing public data available from platforms

As discussed above (subsection 2.2), figures published by micro-work platforms can be a useful starting point, although their interpretation calls for caution. Table 2 presents the figures provided by our selection of platforms. Clearly, international platforms have a much larger user base than national ones. Other differences depend on platforms’ policies: while anyone can sign up to Clickworker or ClixSense in just a couple of simple steps, Appen screens potential new micro-workers after they go through a full-fledged application process, and FouleFactory has closed its registrations, created a waiting list, and put in place a deregistration policy for inactive users. Platforms also differ in their degree of precision: some, such as Microworkers, give very detailed figures and update them regularly, while others, such as Amazon, just provide approximations.

Table 2 mirrors platforms’ practice to communicate the size of their micro-workforce without breaking it down by country. The literature and platforms’ narratives provide more insight. On Amazon Mechanical Turk, French workers represent only a small fraction of the user base (Difallah et al., 2018). On another international platform, Microworkers, France is not among the 10 most represented countries, and remains part of the group of those where less than 20% of total transactions take place (Hirth et al., 2011). On the German platform Clickworker, France and other European countries (excluding Germany) account for a total of 25% of the workforce<fnref n="10" />. No information is available for ClixSense and Appen, and French platforms present a variety of situations: while FouleFactory is only accessible to residents of France, Ferpection has users signing up from France and from the United Kingdom, the United States, Ireland and other countries<fnref n="11" />.

<footnote n="10">¹⁰ See https://www clickworker.com/about-us/clickworker-crowd/ accessed on 3/12/2018</footnote>

<footnote n="11">¹¹ See http://help.ferpection.com/l/fr/les-utilisateurs/combien-y-a-t-il-dutilisateurs-dans-la-communaute accessed on 3/12/2018</footnote>
<table><caption>**Table 2** Number of people signed-up to micro-working sites, worldwide.</caption><thead><tr><td>Platform</td><td>Registered users</td></tr></thead><tbody><tr><td>Amazon Mechanical Turk</td><td>500,000</td></tr><tr><td>Microworkers</td><td>1,215,829</td></tr><tr><td>Clickworker</td><td>1,200,000</td></tr><tr><td>ClixSense</td><td>7,000,000</td></tr><tr><td>FouleFactory</td><td>50,000</td></tr><tr><td>Ferpection</td><td>50,000</td></tr><tr><td>Appen</td><td>1,000,000</td></tr><tr><td>Total</td><td>11,015,829</td></tr></tbody></table>

*Source:* websites of platforms included in analysis (accessed in August 2017 for Amazon Mechanical Turk; September 2018 for all other platforms).

### 3.3 Analyzing experimental data with the “capture-recapture” model

Since Stewart et al. (2015) and Difallah et al. (2018) challenged Amazon’s publicly available data by implementing a “capture-recapture” approach to Mechanical Turk, this method commonly used in bio-ecology and epidemiology has proven to be a valuable tool to estimate the number of micro-workers on a platform. Having posted a micro-task for a relatively long period of time, and allowing repeated participation, the technique consists in “capturing” participants a first time, identifying them as having already done the task, then seeing how many of them get “recaptured” a second time.

We were able to replicate this approach on the French leading micro-working platform FouleFactory. As part of our DiPLab study (see footnote 1), we distributed a questionnaire in the form of a paid task, administered in two steps over two months: the first phase obtained 505 responses and the second 492 responses<fnref n="12" />. While purely logistical issues motivated this two-step collection, we saw it as an opportunity to make an estimate of the number of persons micro-working for Foule-Factory. Indeed, in the second collection, we detected 89 returning respondents - thus enabling us to apply the capture-recapture model.

The capture-recapture model is based on two assumptions: first, the population is closed (that is, nobody entered or left the platform during the study) and second, all subjects have the same chances of being captured. The former was satisfied in our case, as FouleFactory’s closed-registration policy and the short time interval between the two waves limited the number of those entering or exiting the platform. The so-called Lincoln-Petersen formula applied to the data thus collected, with N

<footnote n="12">¹² Calculations after cleaning and filtering of data.</footnote>
the total number to be estimated, $n_1$ the number of users captured in the first wave, $n_2$ the number in the second wave and $m$ the number of recaptured participants, gives :

$$N = \frac{n_1 n_2}{m}$$

Based on the number of individuals observed, the results of the calculation give a population of 2792 microworkers (Table 3). This figure is only 5.6% of the platform’s public declarations, but it is consistent with the domain-specific knowledge of its managers who, answering our questions about the ideal duration of our data collection, were confident that “over two months, [we] can mobilize up to 3000 people on this type of task”.

However, this is likely an underestimation. The second assumption of the model (equal chance of capture) does not apply if users display an uneven level of involvement in micro-work on FouleFactory. When facing the same issue on Amazon Mechanical Turk, Difallah et al. (2018, p. 141) introduce a latent variable (a) that they interpret as propensity to participate: for example, it could reflect the fact that some people perform all the tasks available on the platform, regardless of their appeal, difficulty or remuneration, while others choose tasks more selectively. The initial formula is modified as follows<fnref n="13" />:

$$N^* = \frac{n_1 n_2}{m} \left(1 + \frac{Var(a)}{E(a)^2}\right) = N \left(1 + \frac{Var(a)}{E(a)^2}\right)$$

Difallah et al. (2018) estimated the distribution of a based on 28 observations over two years. In our case, their parameter setting<fnref n="14" /> would give $N^* = 34,166$ (Table 3), a value that still falls short of the public declarations of FouleFactory (50,000 as indicated above), but approximates it if we interpret it as an order of magnitude.

However, there is no guarantee that our population shares the same characteristics of the people who work on Mechanical Turk. We therefore propose an alternative estimate of $N^*$ using a proxy for a, based on one of the questions asked in our survey, about the number of micro-tasks carried out in the previous month. The answers to this question, summarized in Figure 1, show a high diversity of participation - inconsistent with the assumption of equal chances of capture<fnref n="15" />. Using the above formula to calculate $N^*$, we obtain a population of 6531 micro-workers on FouleFactory<fnref n="16" />.

<footnote n="13">¹³ $N \approx N^*$ when the variance of a is close to the mean, but $N < N^*$ otherwise.</footnote>

<footnote n="14">¹⁴ Difallah et al (2018, p. 142) consider that the propensity to participate follows a beta probability distribution, and that therefore the chance to recapture the same subject $n$ times follows a binomial law with parameters $\alpha = 0.29$ and $\beta = 20.9$. With these values, and a sample of $S$ single observations, the population $N^*$ is equal to $\frac{S}{1-f(0|n, \alpha, \beta)}$.</footnote>

<footnote n="15">¹⁵ As our variable was originally a categorical (ordered) one, we have first transformed it into a numerical variable (taking the value of the minimum of the interval in each case: 0, 1, 3, 6, 10) in order to infer a distribution from it, thus enabling calculation of its mean and variance. Alternative transformations of this variable into a numerical one give qualitatively similar results.</footnote>

<footnote n="16">¹⁶ Variations of this calculation, estimating distributions for each possible value in each interval, produce results of a similar order of magnitude (between 5,800 and 8,000).</footnote>
<figure><img src="image_20.png" /><figcaption>**Fig. 1** Number of micro-tasks carried out on FouleFactory in the month preceding the survey, n = 908. Source: Authors' elaboration with DiPLab survey data.</figcaption></figure>

Table 3 summarizes these results. The plain capture-recapture model (Lincoln-Petersen formula) produces an underestimation and its correction (parameters of Difallah et al. (2018)) approximates the number of signed-up users reported by the platform (Table 2). The third estimate, correcting the Lincoln-Petersen figure with a proxy from our DiPLab survey, gives an intermediate figure.

<table><caption>**Table 3** Estimate of the size of the FouleFactory population using the capture-recapture method (basic formula and corrections).</caption><thead><tr><th>Method</th><th>Result</th><th>% of declared (50,000)</th></tr></thead><tbody><tr><td>Lincoln-Petersen formula</td><td>2,792</td><td>5.6%</td></tr><tr><td>Correction with Difallah et al. (2018) parameters</td><td>34,166</td><td>68.3%</td></tr><tr><td>Correction with a proxy from DiPLab survey</td><td>6,531</td><td>13%</td></tr></tbody></table>

*Source:* authors’ elaboration with DiPLab survey data.

The latter estimate is more difficult to interpret based on the information provided so far. We now need to find clues to make sense of it. Also, we need to extend our
analysis beyond the case of one single platform and go back to the full list of the micro-work intermediaries that we have identified as relevant for France (section 3.1).

## 3.4 Analyzing web traffic data to measure the audience of micro-work platforms

We submit that user panels to measure the audience of these platforms, a method borrowed from the study of media but never used in research on platform labor so far, can provide another perspective on the population of micro-workers. For France, we used the service SimilarWeb.com<fnref n="17" />, which makes it possible to estimate the number of unique monthly visitors to micro-working sites worldwide (Table 4, left column).

A potential difficulty that might arise with this type of sources is the double counting that might occur if, for example, the same person connected via two different devices or from two different IP addresses. Although we have no access to SimilarWeb's algorithm, the company ensures that this is controlled for in their calculations. In addition, our own DiPLab survey suggests that most people always connect from the same device and place, typically at home: therefore, this problem is likely negligible.

In addition, given that most platforms have separate web interfaces for client companies and for micro-workers, we can focus exclusively on the latter, namely on the pages dedicated to carrying out the tasks (for example, Fouleurs.com for FouleFactory). We must also exclude simple visitors (also including journalists or researchers like ourselves) from our analysis. To check for this, we look at the average duration of visits, which as per Table 4, is relatively long (central column). This allows us to exclude the presence of simple visitors or internet users who are just searching information, and to argue that these sections of the platforms are almost exclusively accessed by micro-workers. The only exception is Amazon Mechanical Turk, where information from SimilarWeb.com is not entirely reliable as the large number of observed unique visitors even exceeds the platform's own estimate of 500,000 (left column), and must therefore include surfers who are not themselves micro-workers. As discussed above, Mechanical Turk should be seen as a unique rather than a representative case.

Let us now comment especially the right column of Table 4, which provides estimates of the average number of French unique monthly visitors per platform. Starting from FouleFactory, we observe 6,958 unique micro-workers accessing it on average every month – which is very close to the figure of 6,531 that results from

<footnote n="17">¹⁷ The data used by SimilarWeb.com come from two main sources: partnerships that the site has established with Internet service providers and data that the site retrieves with an add-on on users' internet browsers – which provides users with metrics of website traffic in exchange for their browsing data. We have chosen SimilarWeb.com instead of the French market leader Médiamétrie, because the latter sends too few visits to our platforms of interest.</footnote>
<table><caption>**Table 4** Average number of unique visitors on micro-work platforms (left), average duration of visits (center) and average number of monthly unique visitors from France (right).</caption><thead><tr><th>Platform</th><th>Average monthly unique visitors</th><th>Average visit duration</th><th>Average monthly unique French visitors</th></tr></thead><tbody><tr><td>Worker.mturk.com (Amazon Mechanical Turk)</td><td>588,976</td><td>34:19</td><td>1,250</td></tr><tr><td>Microworkers</td><td>174,808</td><td>18:22</td><td>1,835</td></tr><tr><td>Workplace click-worker.com (Clickworker)</td><td>242,579</td><td>7:40</td><td>14,700</td></tr><tr><td>ClixSense</td><td>1,083,000</td><td>11:05</td><td>20,250</td></tr><tr><td>Fouleurs.com (FouleFactory)</td><td>7,647</td><td>26:41</td><td>6,958</td></tr><tr><td>Ferpection</td><td>28,064</td><td>7:10</td><td>8,116</td></tr><tr><td>Appen</td><td>260,699</td><td>5:04</td><td>9,645</td></tr></tbody><tfoot><tr><td>Total</td><td></td><td></td><td>62,754</td></tr></tfoot></table>

Source: Authors' elaboration based on information from SimilarWeb.com, accessed in September 2018. Figures are based on websites' audience tracking over the period July-September 2018.

our corrected capture-recapture calculation as detailed in subsection 3.3. This order of magnitude seems to correspond well to the size of the active population on this platform over a month (as could be measured in the summer of 2018). It should also be noted that almost all of the visits originate from France, in accordance with the platform's policy. The residual cases of people signing in from outside France essentially concern French micro-workers temporarily connecting from abroad — a trend confirmed by responses to our questionnaire. Concerning the other micro-work platforms offering access interfaces in French, ClixSense, Clickworker, Appen and Ferpection are more visited than FouleFactory. This is both due to their open registration policy and to their lifespan (at least in the case of ClixSense, created as early as 2009). Some also benefit from their role as gateways to other micro-work platforms: as noted above, ClixSense gives access to Figure Eight (formerly Crowd-flower, see footnote 7), while Clickworker provides access to UHRS (footnote 9).

To estimate the number of French micro-workers on Amazon Mechanical Turk, where as discussed, SimilarWeb provides only very limited help, we use instead data
from a survey conducted by the team of Panos Ipeirotis at New York University<fnref n="18" />. Extrapolating from the size of the group of French “Turkers” that they observed, i.e. 0.25% of the total, and assuming that their sample is representative, we estimate that the number of French residents operating on Mechanical Turk is around 1,250.

### 3.5 Combining methods to obtain a range of values

The calculation methods we have implemented so far provide differing estimates of the population of micro-workers in France: a high estimate is the number of signed-up users advertised by the platforms themselves (11,015,829 individuals worldwide), a low estimate is the plain capture-recapture model (2,792 individuals in a single platform, FouleFactory) and an intermediate estimate is the audience-based measure (62,754 individuals across platforms all over France).

Each of these approaches has limitations. At least in its basic, uncorrected version, the capture-recapture model underestimates the target population, while the sum of platforms' public declarations overestimates it. The use of audience measurements brings a helpful alternative with intermediate results, but it is based on proprietary techniques whose actual implementation is difficult to audit for external researchers. At this stage, we have no means to choose one single metric from among those three, all the more so as they derive from very different measurement approaches. However, we can use these three results to cross-pollinate each other and help us derive more consistent measures.

Notice, first, that in the case of FouleFactory alone, the two corrections to the capture-recapture model that we have proposed to compensate for the downward bias of the basic Lincoln-Petersen formula (34,166 and 6,531), are close to, respectively, the publicly-released number of registered workers provided by the platform itself (50,000) and audience figures (6,958). These results implicitly validate all these metrics, although they are all lower (to different degrees) than the reference 50,000.

More importantly, audience figures enable an assessment of the number of registered micro-workers who are located in France. To compute their number, we need to extrapolate their frequency of connection to the platforms (calculated from Table 4) as a percentage of the total number of persons signed-up worldwide (Table 2)<fnref n="19" />. Table 5 shows the results of these calculations.

<footnote n="18">¹⁸ Data from the survey "Analyzing MTurk demographics" are available at https://github.com/ipeirotis/mturkDemographics and serve as the basis for the article by Difallah et al. (2018).</footnote>

<footnote n="19">¹⁹ Given the national scope of FouleFactory's activity, we consider all its registrations to be French. For Mechanical Turk we refer to the figures from P. Ipeirotis's survey mentioned in footnote 18. For the other platforms, we take into account the number of visitors from France (Table 4, last column) divided by the total number of visitors (Table 4, first column), then apply the resulting percentage to the number of registrations (Table 2). For example, Ferpection has 8,116 unique visitors from France, out of a total of 28,064 visitors, or 28.92%; multiplying by its publicly declared number of registered users, equal to 50,000, we obtain 14,460.</footnote>
<table><caption>**Table 5** Estimated number of people registered on micro-work platforms and connecting from France.</caption><thead><tr><td>Platform</td><td>Number French</td></tr></thead><tbody><tr><td>Amazon Mechanical Turk</td><td>1,250</td></tr><tr><td>Microworkers</td><td>12,766</td></tr><tr><td>Clickworker</td><td>72,720</td></tr><tr><td>ClixSense</td><td>130,900</td></tr><tr><td>FouleFactory</td><td>50,000</td></tr><tr><td>Ferpection</td><td>14,460</td></tr><tr><td>Appen</td><td>37,000</td></tr><tr><td>Total</td><td>319,096</td></tr></tbody></table>

Source: Authors' elaboration.

On this basis, we can go a step further and attempt to generalize the capture-recapture model initially used just on one platform, by applying its ratios (Table 3, last column) to the total number of French micro-workers we have just estimated (319,096). We thus obtain a range, with a low estimate of 17,869 (5.6% of the declared total for France), a high estimate of 217,943 (with a “corrected” rate of 68.3%) and an intermediate estimate of 41,482 (with the alternative correction of 13%).

## 3.6 Accounting for users who micro-work on several platforms

We cannot rely on these three estimates until we have made an additional correction to take into account workers performing micro-tasks on several platforms. As discussed above, disregarding this type of multi-activity amounts to over-estimating the number of workers by counting the same persons multiple times. How many platforms do people use at the same time? To answer this question, let us turn to our DiPLab survey, in which participants were asked to indicate their use of different micro-work platforms from a list of options. 16.6% of them use at least two platforms from among the 7 included in our sample, which amounts to say that on average, each person uses 1.27 micro-tasking platforms.

Assuming our sample is representative, and extrapolating its behavior to the entire French micro-working population, amounts to applying this rate to the measures previously obtained (Table 6).

Little is known of micro-workers' multi-activity in other contexts but, comparing our survey to studies of other online practices such as the buying and selling of goods and services (where for example, Oxera (2015) reports an average of 2.2 platforms per user), it appears that our rate of multi-activity is low. This is somewhat
<table><caption>**Table 6** Estimate of the population of micro-workers refined to account for use of multiple platforms.</caption><thead><tr><th>Range</th><th>Measure</th><th>Result (reminder)</th><th>Result (minus multi-activity)</th></tr></thead><tbody><tr><td>Low</td><td>Uncorrected capture-recapture</td><td>17,869</td><td>14,903</td></tr><tr><td>Intermediate</td><td>Audience measure</td><td>62,754</td><td>52,337</td></tr><tr><td>High</td><td>Number of declared users (France)</td><td>319,096</td><td>266,126</td></tr></tbody></table>

Source: Authors’ elaboration.

surprising as the globally competitive context of micro-work, together with systematic excess supply of labor (subsection 2.2) would lead to expect higher rates. The DiPLab survey suggests that practical impediments play a role. Some respondents hint that entry costs of maintaining active accounts on several platforms are relatively high, both in terms of incompatibility of payment systems (for example, FouleFactory uses Mango Pay while Microworkers uses Paypal, Skrill and Payoneer) and in terms of limited portability of “qualifications” (unpaid tests needed to access some types of micro-tasks, for example tasks that require knowledge of a specific language). Additionally, the English language used on many international platforms puts off some French micro-workers, even though some of their tasks are in French.

## 4 How to interpret results as indicators of micro-workers' level of activity?

We now turn to the meaning of these different measures of the size of the French micro-working population. Building on the qualitative evidence collected via 92 semi-structured interviews conducted within the DiPLab project, we associate our figures to characterizations of users according to their level of activity. We distinguish a group of “very active” (subsection 4.1), a group of “regular” (subsection 4.2) and one of “occasional” (subsection 4.3) micro-workers.
## 4.1 Very active micro-workers

Our first, lower estimate of 14,903 micro-workers is based on the uncorrected capture-recapture model, first applied to FouleFactory, subsequently extrapolated to the total number of French registrations over the seven platforms of interest, and finally corrected to remove duplicates (subsection 3.3). As discussed, the Lincoln-Petersen formula on which this figure is based, is downward biased when the chances of capture (here, levels of engagement with the platform) are unequal, and we can add now that it tends to over-sample users with a high level of activity. Indeed, the number of FouleFactory users who were captured and then also re-captured is remarkably higher than we would have predicted (Stewart et al., 2015; Difallah et al., 2018).

These remarks suggest that this figure should not be simply dismissed as an under-estimate, but provides instead a rough guess of the number of what we might call “very active” micro-workers. While we do not have elements to characterize them in much detail, we can say at least that they connect to micro-working platforms at least weekly – as 90% respondents to our sample do. We also have examples from our qualitative fieldwork of what high activity may mean in practice. In addition to her primary occupation as a nurse, one respondent performs “simple and quick tasks” every day during her breaks. Another, expressing her “fear of missing out tasks”, even goes so far as leaving FouleFactory continuously open on her computer.

## 4.2 Regular micro-workers

The estimate we obtain through audience measurements from the seven micro-work platforms considered is much larger, with 52,337 users (average monthly number of unique visitors from France, calculated over three months, summed across platforms (section 3.4) and corrected downward to account for workers who switch across platforms (section 3.6)). Unlike the above, this method is more likely to over-sample users who connect at least monthly – indeed any more frequent activity won’t change the total, and conversely more sporadic connections are less likely to be counted at all. Thus, we can take this measure as an estimate of the number of micro-workers with an intermediate level of activity, and we label it “regular”. It likely involves using platforms for a secondary source of income.

For a more concrete sense of what it means in practice, let us resort again to our DiPLab interviews. An editorial manager in a communication agency has been micro-working for three years. She signed up during a period in which her “lifestyle required an additional income”, resulting in “a compulsive need to earn more money”. Since then, she has been routinely returning to micro-work, though not with the same intensity as before, and doing essentially questionnaire-based micro-tasks that she sees as an extension of her education. This case suggests that “regular” micro-workers often have a selective attitude towards tasks, and just won’t
perform any job that they see online. Notice, also, that users may move from very active to regular (and presumably, the other way round) at different times.

## 4.3 Occasional micro-workers

Our last estimate features 266,126 persons and corresponds to registered users as declared by platforms themselves, after removing non-French users and duplicates. Because these are just users who signed up with one or more micro-work platforms, in principle they may have any level of activity. But overall, people with low levels of activity will be much more numerous within this group. Thus, we can expect the great majority among them to be “occasional” micro-workers, who perform less than one task per month. Some of them may do tasks very rarely, and there might even be cases of users who never did any tasks at all. Using a corrected measure of the number of registered users is the only way to observe the long tail of this population that connects less frequently, is more difficult to reach through platform-based dedicated surveys (like the one we launched), and may escape web traffic panels.

Our interviews suggest that broadly speaking, two sub-populations can be distinguished within the "occasional": One includes users who signed up out of curiosity and quickly abandoned micro-work, possibly because of the fluctuating availability and unsuitability of tasks – two points often emphasized by our study participants. They never de-registered, but their use remains sporadic. Another includes individuals who were “very active” or “regular” at some point in time, have currently reduced their involvement with these platforms, but may get back to them again in the future depending on their circumstances.

One respondent, holding a day-job as administrative secretary, illustrates this situation well. He signed up to FouleFactory in 2016, but “quickly dropped out”. A few months later, he received an email indicating that the platform would delete his account due to inactivity. He then decided to resume his micro-working activity in a relatively sustained way. He now claims he earns about 80 euros a month from micro-tasks, almost four times more than the average in our survey (Casilli et al., 2019). Another micro-worker, currently seeking employment, also alternates between periods of inactivity and of intensive performance. In her working life, she has always experienced alternating phases of unemployment and employment. She says she stops micro-work as soon as she finds a job because she “no longer has time”, while her phases of unemployment correspond to periods of intensive micro-work because she prefers to “do this rather than nothing”.
# 5 Discussion: strengths and limitations of present methods

Our three estimates respond to the need of assessing the size of a workforce that escapes the definitions and measurements of official statistics. Rather than opting for the relatively safe and simple option of focusing on a single platform, we look at one country as the basic unit at which any policies can be devised, but where multiple platforms operate simultaneously. We fully take into account the specificities of micro-work, notably diversity in levels of engagement, with many people practicing it as a secondary or infrequent activity; absence of geographical anchor, which implies co-existence of both national and international platforms; and, in an attempt to bridge the digital labor and the two-sided-markets literatures, multiple use of different platforms. Our choice of methods borrows from some of the most advanced literature on micro-work platforms (the capture-recapture model) and from neighboring fields (web traffic measures) to bring more precision to publicly available data from companies. Our mix of methods enables us to do comparisons and quality checks, to obtain more reliable results.

## 5.1 Methodological limitations

The methods we have deployed were a journey into uncharted territory, and required us to make some strong assumptions and simplifications. Lack of data restricted the set of platforms we could study. The capture-recapture method was applied to only one platform, FouleFactory, and we extrapolated its basic result (the uncorrected Lincoln-Petersen formula) to all others, assuming they would exhibit the same behavior. What's more, our survey included just two waves of capture and recapture, allowing only for approximate corrections of the downward bias resulting from the equal-capture-chances assumption. Regarding audience measures, constructed by private companies for business purposes, there was very limited scope for us to control their quality. Finally to assess the prevalence of micro-workers operating on several platforms, we had to assume our FouleFactory survey sample was representative of the French micro-working population.

For all these reasons, we believe that *our figures should be interpreted as orders of magnitude rather than exact amounts*: we will retain about 15,000 “very active”, 50,000 “regular” and 260,000 “occasional” micro-workers. Because these figures have been obtained through different methods, they cannot meaningfully be added; neither can they be construed as subsets of one another, the “very active” constituting an inner circle and the “occasional” an outer circle, with the “regular” in-between. These labels only indicate the levels of engagement that each of these estimates over-represents in comparison to the others, but we cannot exclude overlaps.
## 5.2 Why our methods are nevertheless reliable

Despite these limitations, these figures provide helpful indicators for research and policy. Moreover, the reliability of our methods lies in their consistency with existing evidence. The 15,000 can be seen as a sort of lower bound on the number of French micro-workers: there are at least as many people who perform this activity. At a very basic level, it is evidence that there are micro-workers in France at all: even if cost optimization may lead companies to search for cheaper online labor overseas as discussed in section 2.2, and even if many large data orders are perhaps more likely to transit through English-language international platforms, there is still one fraction of micro-work that is done in the country, and people who are available to do it. We have interpreted this estimate as inclusive of mostly very active people, which means the volume of economic activity they generate is roughly comparable to that of French Uber drivers, whose number was estimated at about 14,000 in an aforementioned IGAS report (section 3.2, Amar and Viossat (2016)), and of Deliveroo couriers, who are 9,300 couriers in France if we trust company figures (Deliveroo, 2018). If these more high-profile cases have attracted a lot of academic and policy attention, micro-work then deserves the same.

The 50,000 is an intermediate level, probably the most representative of the widespread trend to do platform work as a secondary or accessory activity, requiring regularity but not necessarily high frequency. The volume of economic activity that these workers generate is presumably small, but not negligible. Use of platforms for extra income rather than a main job is not unique to micro-work, and has already been observed in surveys of platform labor in general (Huws et al., 2017; Pesole et al., 2018; Urzì-Brancati et al., 2019; O'Farrel and Montagnier, 2019). Regarding micro-work specifically, this usage may be specific to Europe and North America, with providers in low-income and emerging countries choosing instead to micro-work full-time (or almost so) to meet growing demand by large digital companies (Schmidt, 2019). In countries such as France, the need for an extra source of income that the existence of these workers brings to light, calls for a better understanding of the conditions under which it emerges: what are the unmet needs that push people to seek additional resources through micro-work?

The 260,000 corresponds to the number of French users registered on one or more of the seven micro-tasking platforms considered. This might seem a surprisingly large number, especially for a type of platform work that has attracted limited attention from the general public so far. However, it is the same order of magnitude as the estimate of about 300,000 that might be derived from the 2018 Eurobarometer, and lower than the 900,000 of the European COLLEEM survey, as reported in subsection 2.1 (although these comparisons are only lightly indicative because definitions vary across these studies). Also, its full-time equivalent may be very small, because not all of these 260,000 do a lot of tasks<fnref n="20" />. The real question is what this

<footnote n="20">²⁰ A back-of-the-envelope calculation suggests that, with micro-workers earning an average of 20 euros per month (a result from our DiPlab survey, see subsection 4.3), and a monthly net minimum wage set at around 1,230 euros in 2018, then 260,000 occasional micro-workers would correspond</footnote>
# 6 Acknowledgments

This paper is based on results of the “Digital Platform Labor” (DiPLab) project, co-funded by Maison des Sciences de l'Homme Paris-Saclay; Force Ouvrière, a workers' union, as part of a grant from Institut de recherches économiques et sociales (IRES); and France Stratégie, a service of the French Prime Minister's office. The platform FouleFactory/Wirk offered logistical support, and Inria provided complementary funding. For more information: http://diplab.eu/

The authors would like to thank current and former DiPLab team members, notably Maxime Besenval, Odile Chagny, Marion Coville, Touhfat Mouhtare, Lise Mounier, Manisha Venkat and Elinor Wahal, for their inputs to the project in general, and this paper in particular.

A preliminary version of this paper was posted on arXiv (n:1901.03889 [econ.GN]) and in French, as a working paper in the i3 series<fnref n="21" />. We thank Mark Graham, Mary Gray, Lilli Irani, Emmanuel Lazega, Vili Lehdonvirta, Antoine Naboulet, Etienne Ollion, Elise Penalva-Icher, Maurice Ronai and Alex Wood for useful and constructive feedback. Any remaining errors or inconsistencies are exclusively ours.

# References

* Amar, N. and Viossat, L.-C. (2016). Les plateformes collaboratives, l'emploi et la protection sociale. Report IGAS n. 2015-121R, IGAS.

* Athey, S., Calvano, E., and Gans, J (2018). The impact of consumer multi-homing on advertising markets and media competition. *Management Science*, 64(4):1574–1590.

* Berg, J., Furrer, M., Harmon, E., Rani, U., and Silberman, M. (2018). Digital labour platforms and the future of work: Towards decent work in the online world. ILO Report.

* Casilli, A. (2017). Digital labor studies go global: Toward a digital decolonial turn. *International Journal of Communication*, 11(0).

* Casilli, A., Tubaro, P., Le Ludec, C., Coville, M., Besenval, M., Mouhtare, T., and Wahal, E. (2019). Le micro-travail en France. Derrière l'automatisation, de nouvelles précarités au travail ? Report of the Digital Platform Labor (DiPLab) project.

* Deliveroo (2018). Deliveroo lance marketplace+. *Deliveroo France Newsroom*.

* Difallah, D., Filatova, E., and Ipeirotis, P. (2018). Demographics and dynamics of Mechanical Turk workers. In *Proceedings of WSDM 2018: the Eleventh ACM International Conference on Web Search and Data Mining*, pages 135–143. ACM.

<footnote n="21">²¹ URL: http://i3.cnrs.fr/wp-content/uploads/2019/02/WPi3-19-SES-02-LeLudec-Tubaro-Casilli.pdf</footnote>
Eichhorst, W., Hinte, W., Rinne, U., and Tobsch, V. (2017). How big is the gig? assessing the preliminary evidence on the effects of digitalization on the labor market. *management revue - Socio-Economic Studies*, 28(3):298-318.

Forde, C., Stuart, M., Joyce, S., Oliver, L., Valizade, D., Alberti, G., Hardy, K., Trappmann, V., Umney, C., and Carson, C. (2017). The social protection of workers in the collaborative economy. Report for the committee for employment and social affairs, European Parliament.

Fort, K., Adda, G., and Cohen, K. (2011). Last words: Amazon Mechanical Turk: Gold mine or coal mine? *Computational Linguistics*, 37(2):413-420.

Frenken, K. (2018). Unpacking the gig economy – are we asking the right questions? Presentation at the Reshaping Work 2018 Conference, Amsterdam, 25-26 October.

Fundación Felipe González and University of Hertfordshire (2019). Huella Digital: La plataformización del trabajo en Europa. Ficha informativa de España (report of the Digital Footprint comparative project).

Graham, M. and Anwar, M. (2019). The global gig economy: Towards a planetary labour market? *First Monday*, 24(4).

Graham, M., Hjorth, I., and Lehdonvirta, V. (2017). Digital labour and development: impacts of global digital labour platforms and the gig economy on worker livelihoods. *Transfer: European Review of Labour and Research*, 23(2):135-162.

Gray, M. and Suri, S. (2017). The humans working behind the AI curtain. *Harvard Business Review*, (9):2-5.

Hirth, M., Hoßfeld, T., and Tran-Gia, P. (2011). Anatomy of a crowdsourcing platform - using the example of Microworkers.com. In *2011 Fifth International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing*, pages 322-329.

Huws, U., Spencer, N., Syrdal, D., and Holts, K. (2017). Work in the european gig economy: Research results from the UK, Sweden, Germany, Austria, The Netherlands, Switzerland and Italy. Commissioned report, Foundation for European Progressive Studies.

Irani, L. (2016). The labor that makes AI "magic". AINow Summit, White House and NYU.

Kässi, O. and Lehdonvirta, V. (2018). Online Labour Index: Measuring the online gig economy for policy and research. *Technological Forecasting and Social Change*, 137:241-248.

Katz, L. and Krueger, A. (2019). The rise and nature of alternative work arrangements in the united states, 1995-2015. *ILR Review*, 72(2):382-416.

Keith, M., Tay, L., and Harms, P. (2017). Systems perspective of Amazon Mechanical Turk for organizational research: review and recommendations. *Frontiers in Psychology*, (8):1359.

Kuek, S., Paradi-Guilford, C., Fayomi, T., Imaizumi, S., and Ipeirotis, P. (2015). The global opportunity in online outsourcing. Report, World Bank.

Mayer-Schönberger, V. and Cukier, K. (2013). *Big Data: A Revolution That Will Transform How We Live, Work, and Think*. Houghton Mifflin Harcourt.
Mrass, V. and Peters, C. (2017). Crowdworking-Plattformen in Deutschland. Leimeister, J.M. (ed.), Working paper series, nr. 16. Kassel, Germany.

O'Farrel, R. and Montagnier, P. (2019). Measuring platform mediated workers. OECD Digital Economy Papers, n. 282.

Ojanperä, S., Graham, M., Straumann, R., De Sabbata, S., and Zook, M. (2017). Engagement in the knowledge economy: Regional patterns of content creation with a focus on sub-saharan africa. *Information Technologies and International Development*, (13):33–51.

Oxera (2015). Benefits of online platforms. Report prepared for google.

Pesole, A., Urzì-Brancati, C., Fernández-Macías, E., and González-Vázquez, I. (2018). Platform workers in Europe: Evidence from the COLLEEM survey. Report EUR 29275 EN, Publications Office of the European Union, JRC112157.

Robinson, J., Rosenzweig, C., Moss, A., and LItman, L. (2019). Tapped out or barely tapped? recommendations for how to harness the vast and largely un-used potential of the mechanical turk participant pool. PsyArXiv preprint, doi: org/10.31234/osf.io/jq589.

Schmidt, F. (2019). Crowdproduktion von Trainingsdaten: Zur Rolle von Online- Arbeit beim Trainieren autonomer Fahrzeuge. Report, Hans-Böckler-Stiftung.

Stewart, N., Ungemach, C., Harris, A., Bartels, D., Newell, B., Paolacci, G., and Chandler, J. (2015). The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers. *Judgment and Decision Making*, 10(5):479–491.

ter Weel, B., van der Werff, S., Bennaars, H., Scholte, R., Fijnje, J., Westerveld, M., and Mertens, T. (2018). De opkomst en groei van de kluseconomie in Nederland. SEO Report n. 2018-30.

Tubaro, P. and Casilli, A. (2019). Micro-work, artificial intelligence and the automotive industry. *Journal of Industrial and Business Economics*, 46(3):333–345.

Urzì-Brancati, C., Pesole, A., and Fernández-Macías, E. (2019). Digital labour platforms in Europe: Numbers, profiles, and employment status of platform workers. Report EUR 29810 EN, Publications Office of the European Union, JRC117330.
# Who are the micro-workers behind AI?

Paola Tubaro

## Abstract

Micro-workers undertake menial online tasks to generate and annotate data for machine learning, to correct and verify the outputs of artificial intelligence systems, and to replace algorithms when they fail. This chapter examines what is micro-work and introduces some of the real people who carry out micro-tasks.

**Key words:** Micro-work, platform economy, digital labor, artificial intelligence, working conditions.

## 1 Introduction

Real men and women contribute to producing artificial intelligence (AI) solutions through “micro-work” (Irani, 2016; Schmidt, 2019; Tubaro and Casilli, 2019). Data-related jobs are fragmented into myriad small tasks that can be performed remotely online, and specialized digital platforms such as Amazon Mechanical Turk, Microworkers and Clickworker allocate them to anonymous providers who work behind their computer or smartphone screens. Such tasks may consist in labeling images, for example by drawing a box around, and/or adding a tag to, each car in a picture of street traffic. Other tasks are textual, such as transcription of license plates and shopping receipts, or translation of short sentences. Others are about content moderation, for example flagging derogatory words in a web page (Roberts, 2019). Quick and repetitive, these tasks are typically paid on a piecework basis, with rates as low as a few cents. Their silent but essential contribution to the functioning of smart equipment, computer vision systems and voice assistants, is what Gray and Suri (2019, 2017) call the "paradox of automation's last mile": smart technologies

<footnote>Laboratoire de Recherche en Informatique, Centre National de la Recherche Scientifique, Paris Saclay, e-mail: paola.tubaro@lri.fr</footnote>
incessantly create a need for so-called “human-in-the-loop” contributions, so that the final outcomes result from a mix of machinery and people.

Micro-work is an instance of digital platform labour, whereby workers are not employees but sub-contractors usually paid by piece-rate, subject to algorithmic control performed by the platform (Wood et al., 2019). Because it is mostly performed online, it differs from geographically sticky (Graham et al., 2017) "gig" work such as urban transportation through Uber and food delivery through Deliveroo. Neither is it the same as *online freelancing*, which is also performed online but concerns more creative work such as design and software development, involves qualified professionals, and entrusts them with full projects rather than single tasks (Schmidt, 2017). Micro-work pushes to the extreme the “datafication” processes that underlie all these activities.

Whose work is it? Who are the people behind today's AI, what motivates them to engage in this activity, and what are their working conditions? In what follows, I show that they are often well-educated and professionally active but unable to earn enough, sometimes owing to care duties. The unregulated, unprotected setting of digital platform work is little suited to improve their chances, especially for the most vulnerable among them. Yet, they are indispensable to AI production. They prepare data for the machine learning algorithms that sustain today's AI solutions, check the outputs of AI systems for accuracy and correct them if necessary, and even impersonate AI when algorithms fail. Their contribution is needed at all stages of AI development and commercialization and therefore, it is not a temporary stopgap measure but a systemic input. Thus, credible commitment to responsible, ethical AI in our economy and society requires considering micro-working conditions.

To support these claims, I combine insights from the nascent international literature on micro-work with empirical results of a mixed-methods research on micro-work in France, DiPLab (“Digital Platform Labor”). The study comprised a questionnaire distributed through a local micro-work platform, Foule Factory<fnref n="1" />, and in-depth interviews with micro-workers, platform owners, business clients and other stakeholders (Casilli et al., 2019). As a highly industrialized country and a pioneer in information technologies, France is exemplary of more general trends, while language and some other specificities allow comparing and contrasting it to other known cases studied in the literature which, so far, has privileged English-speaking settings.

## 2 Who are the micro-workers?

Let us first discover the identity of the people who engage in micro-work. Where are they located? Are they men or women, young or old, more or less educated? What else do they do in life, and why do they micro-work?

<footnote n="1">¹ “Foule” means “crowd” in French.</footnote>
## 2.1 Low income vs industrial countries

The first attempts to describe the micro-working population date back to 2010, just a few years after the international launch of Amazon's pioneer platform Mechanical Turk (Ipeirotis, 2010; Ross et al., 2010). Its demographics were less skewed at the time, with about 50% US and 40% India residents, while these percentages are now 75% US and about 16% India, the rest consisting of negligible proportions from third countries (Difallah et al., 2018). However, this does not mean that there are no micro-workers elsewhere. Hirth et al. (2011) analyzed the platform Microworkers and found it much more geographically diverse than Amazon Mechanical Turk, with 60% of the population coming from Bangladesh, India and the US together. More recently, Posch et al. (2018) surveyed the platform Figure Eight (previously called Crowdflower and now part of Appen, an international giant in data services for machine learning and artificial intelligence) to find, again, significant geographical diversity. A study commissioned by the European Parliament found European micro-workers on four international platforms (Mechanical Turk, Clickworker, Figure Eight and Microworkers) (Forde et al., 2017).

The first conclusion, then, is that micro-work is a global phenomenon, although different platforms may have different country compositions. For this reason, it is important not to limit our analysis to just international platforms, but to take into account local ones too: in the setting of this study, France, one of them is Foule Factory, a French-language platform that serves the needs of local clients through local workers.

## 2.2 Not only millennials

In an early report on worldwide online labor (including both qualified freelancing and micro-work), Kuek et al. (2015) show that most workers are “millennials” (born in the 1980s and 1990s), especially in emerging countries such as India. However, later studies suggest that the youngest among them, including students, are not the majority. If 60% of Forde et al. (2017)'s respondents (across four platforms and throughout Europe, see sub-section 2.1) were below 40 years of age, a long-term demographic survey of Amazon Mechanical Turk finds that 65.5% of the sample are 25-44 years old (Difallah et al., 2018)<fnref n="2" />.

The DiPlab study comes to similar results: all workers are 18 years or older, due to the conditions imposed by the Foule Factory platform that exclude under-age users, and only one person in the sample is over 75. Most respondents (63.4%) are 25-44 years old – in line with the just-cited result of Difallah et al. (2018). This age range is clearly over-represented relative to the general French population, as can be observed by comparing this result to figures by INSEE, the National Statistical

<footnote n="2">² For comparison purposes, I have independently re-analyzed the data used in this study, available online at https://github.com/ipeirotis/mturk\_demographics</footnote>
Institute, which indicate that this age range amounts to only 35% of the general French population of age 18-74³.

<table><caption>**Fig. 1** Age distribution of DiPLab survey participants, compared to the general French population (in percentage). Interpretation: 33.4% of DiPLab respondents are 25-34 years old. Source: adapted from Casilli et al. (2019).</caption><thead><tr><th></th><th>DiPLab</th><th>France</th></tr></thead><tbody><tr><td>18-24</td><td>12.1</td><td>11.7</td></tr><tr><td>25-34</td><td>33.4</td><td>17</td></tr><tr><td>35-44</td><td>30</td><td>18</td></tr><tr><td>45-54</td><td>15.2</td><td>19.5</td></tr><tr><td>55-64</td><td>6.3</td><td>18.2</td></tr><tr><td>65-74</td><td>3</td><td>15.6</td></tr></tbody></table>

Micro-work, then, is not a student job but an activity for people whose age is usually associated to having completed even long studies, having a job and a (potentially young) family. Let us then look at other variables to better understand their situation.

## 2.3 Not only men

Are micro-workers more women or men? On Amazon Mechanical Turk, Difallah et al. (2018) observe about equal participation of men and women, although percentages differ greatly across countries : in particular, women are about 55% of US “Turkers”, but are much less numerous than men in India. Throughout Europe, Forde et al. (2017) count 60% men. A recent ILO research report that surveyed five international platforms (Mechanical Turk, Clickworker, Figure Eight, Microworkers and Prolific Academic) finds a more extreme gender imbalance, with only one third of workers in industrial countries being women, and barely one fifth in low-income and emerging countries (Berg et al., 2018).

Yet women are slightly over-represented in the DiPLab survey of French micro-workers, comprising 56.1% of the Foule Factory population – close to the percentage of American women active on Mechanical Turk reported above. Ferpection, another French platform and one of the very few that publicly provide figures about their workforce, says it has 57% women.

<footnote n="3">³ INSEE, Demographic state of France 2018. URL: https://www.insee.fr/fr/statistiques/1892086?sommaire=1912926.</footnote>
<figure><img src="image_21.png" /><figcaption>**Fig. 2** Gender distribution of DiPLab survey participants (platform Foule Factory, 2018), compared to: 1/ the general French population aged 18-74, observed by INSEE (2018); 2/ proportions displayed by the French platform Ferpection in 2018; 3/ data collected by the team of Difallah et al. (2018) on Amazon Mechanical Turk across countries; 4/proportions found in the survey of Forde et al. (2017). Interpretation: 56.1% of participants in the 2018 DiPLab survey are women.
Source: adapted from Casilli et al. (2019).</figcaption></figure>

It is not true, then, that micro-work is a predominantly male activity. Rather, women and men choose different platforms. There is a majority of women on Ferpection and Foule Factory, two local, French-language platforms; but among the Mechanical Turk workers from France surveyed by Difallah et al. (2018), only about 25% are women. This depends largely on platforms' history and policies: Mechanical Turk restricted access of non-US residents around 2013, so its French users are early adopters, more likely to be men as with most digital technologies (Borgonovi et al., 2018). Conversely, sign-ups to Foule Factory boomed after a 2015 TV report that presented it as an earning opportunity for all. In passing, it is for this reason that it is important not to limit analysis to the most high-profile international platforms, as most of the literature has done to date, missing important parts of the population.

Perhaps more importantly, country of residence changes the role of gender with respect to micro-work. Why do women micro-work – when they do? Ipeirotis (2010) consider that more male workers from India participate on Amazon Mechanical Turk because it is their primary source of income, while more women from the US participate because they consider Mechanical Turk a secondary source of income. Is this a more general difference between emerging and industrial countries? Using the case of France to provide a clue, what follows looks at the extent to which DiPlab respondents use micro-work as a complementary activity, in relation to their family and employment status.
## 2.4 Highly educated workers, in work, with family

Kuek et al. (2015) was one of the first studies to show that online workers, including micro-workers, are more likely to hold a University degree than the general population – a finding that almost all subsequent literature confirms. DiPLab is not an exception: 66.5% of the surveyed micro-workers have a bachelor’s degree or higher, in comparison to just 27.8% of the general French population. It is certainly ironic that highly qualified people do tasks such as tagging objects in images or transcribing sentences.

As expected (sub-section 2.3), micro-work is more often practiced as a complement to a full- or part-time job, especially in industrial countries. Only 52% of the workers surveyed worldwide by Berg et al. (2018) use micro-work as a secondary source of income, against 68% of Forde et al. (2017)'s European micro-workers and just over 60% of French DiPlab respondents (which, for just over half of them, is a clerical occupation). Of the 40% French DiPLab micro-workers who have a long-term employment contract, more than two thirds work full-time. As in the general population, women are more numerous to work part-time (32%, against 11% of men). Women are more numerous (about 60%) than men (slightly less than half) to depend at least partly on their spouse or partner for their income.

Yet even in industrial countries, micro-work is sometimes practiced as a remedy to inactivity. 36% of the French micro-workers surveyed by DiPLab are inactive, that is, they have no (main or secondary) professional activity aside from their micro-tasks, but only 15% of them are retired. A small number (mostly men) are students and the others are in diverse situations, but always with a majority of women. For these people, micro-work is an essential source of income.

Regarding family status, DiPlab data show that a larger number of women have children compared to men (55% vs 39%). Women also devote 6 to 12 hours per week more than men to domestic chores.

Such imbalances help to answer the question of why women micro-work (sub-section 2.3): mostly in working age but more numerous to work part-time (or to be inactive) than men, more frequently depending on their spouse for income, more often responsible for family care, and spending more time at home, they use all their free time and all their breaks between work and home activities to do online micro-tasks. Micro-work provides some welcome additional earnings, but also adds to the burden of formal employment and care work – eating into their leisure time.

## 2.5 A lower-income population that needs extra earnings

With these situations, are micro-workers poorer than the general population? To answer this question, reference can be made to the French Observatory of Inequalities which defines the lower classes as the 30% of the population with the lowest incomes. Among the surveyed DiPlab micro-workers, the lower classes are largely over-represented, with 51% meeting this criterion.
An alternative criterion is the poverty threshold, computed as 50% of the coun-
try's median income<fnref n="4" />. 22% of the micro-workers under study live under this thresh-
old, compared with only 8% of the general French population.

Against this background, online micro-tasks are an attempt to cope. The DiPlab survey asked respondents three reasons why they micro-work. The great majority say they need money – either as their first, second or third answer (Figure 3). In interviews, some said they actively look for better-paid tasks and feel frustrated if these turn out to be less rewarding than expected.

<figure><img src="image_22.png" /><figcaption>**Fig. 3** Reasons for micro-work. Each DiPLab survey participant was invited to choose three of them, and rank them in order of importance. Interpretation: 44.93% choose “I need money” as the first reason why they micro-work. Source: adapted from Casilli et al. (2019).</figcaption></figure>

If micro-work is a necessary – albeit often complementary – income source, does it provide enough for workers to support themselves? Let us now look at their earnings from this activity.

2.6 How do micro-workers fare?

Earnings from micro-work are low. On Amazon Mechanical Turk, the median hourly remuneration is only about US$ 2 (Hara et al., 2018). The figures of Berg et al. (2018), calculated across five different international platforms, are similar (US$ 2.16/hour). One reason why remunerations are so low is that a lot of time is spent in unpaid activities, such as searching from the list of available tasks and passing qualification tests (for example, language proficiency tests) to have access

<footnote n="4">⁴ This level is currently estimated at 855 euros per month for a single person. Source: INSEE and French Observatory of Inequalities.</footnote>
to a wider range of tasks. Hourly rates are higher when only paid work is taken into account.

But hourly rates do not account for variations in the number of hours worked – which are huge because micro-work is often a complement rather than a main earning activity (sub-section EducationWork). Monthly, rather than hourly, earnings are better indicators in this respect: DiPLab respondents earn 22 euros a month on average, although the distribution is very skewed with a small number of people who make up to 2,000 euros a month, sometimes through multi-activity over different platforms (as do over 80% of DiPlab respondents).

In sum, people in different life situations micro-work. They live both in industrial and in low-income countries, they are both men and women, both with and without a primary professional activity beyond micro-tasking. What is common to many of them is a high educational level, somewhat at odds with the typically repetitive nature of micro-tasks. It is also the need for an extra source of income, either because they experience financial difficulties or because care duties prevent them from seeking better-paid employment opportunities. However, most people earn very little from micro-work, and their engagement is highly variable. Why? To answer this question, it is now time to look more closely at what micro-workers do.

# 3 What do micro-workers do?

Let us first review the activities that micro-workers do, to better understand the challenges that they face, before looking at the industrial needs they help to meet.

## 3.1 What are micro-working tasks?

Among other things, the DiPLab survey asked micro-workers to briefly describe the last task they had performed online. Although a single question at a specific point in time cannot provide an exhaustive description of all possible tasks, it can give a concrete idea of what the daily activities of a micro-worker consist in. Let us first review the most frequently mentioned answers, before trying to make sense of them.

A common task consists in responding to online surveys, to the satisfaction of many micro-workers who find them clearly presented and relatively well-paid, although others dislike their length. Another task reported by one in five DiPLab respondents involve voice: reading aloud a few short sentences in French and audio-recording them. Variants of this task include requests to record five different ways to say, ask what the weather is like. Other recurring tasks are about recognition of objects in images: one of them asked micro-workers to draw bounding boxes around vegetables (tomatoes, carrots, etc.) in pictures of salads. There are also document management tasks such as CV anonymization, reported by one fourth of respon-
dents. According to Claire<fnref n="5" />, this task was about “removing all distinctive marks that could be discriminatory”, but it also required tagging the spaces in the document where names, birth dates, addresses and other personal or sensitive information were placed originally. Other examples of document management tasks involve classifying objects such as DVD titles, sorting photos, or recognizing whether commercial products found in different online marketplaces are actually the same.

It is useful to extend our gaze beyond the questionnaire, to consider additional types of tasks that did not surface there, but were mentioned in in-depth interviews. One is about moderating videos, text and images circulating on the web, in order to remove violence, pornography, drugs, and other inappropriate material. Aline said that, after the attacks of 2015-16, she checked “monstrous” terrorist content for several weeks, full-time, as clients “were panicking” for fear that gore images would flood their websites.

Another task that was mentioned in interviews, consists in checking the accuracy of speech recognition by voice assistants. Juliette spent several months comparing automated transcriptions and audio recordings of users’ conversations with their virtual assistant, check them for accuracy, and if needed, rectify them. Other interviewees checked the accuracy of automated transcriptions of scanned documents such as commercial receipts and invoices, adding corrections as appropriate.

Other interviews paint a more sinister picture. Kader, an entrepreneur, derided his competitors who allegedly made Madagascar “the French AI leader”. Why? Because, he argued, they claim they do AI while in reality, they just recruit cheap micro-workers in low-income, French-speaking African countries to do the work manually. Some micro-workers steal robots’ jobs, so to speak – for example when they have to infer clients’ tastes from social media and recommend some products to them.

## 3.2 Why tasks are challenging after all

However sketchy, this list of tasks helps us understand why micro-work is not necessarily straightforward – and why some workers engage with it very little. Routine tasks such as recording one’s voice or looking for tomatoes in pictures are relatively light to perform. Although they are often perceived as “silly”, as Anne said, some workers are just happy with the little pay they get for them, given the limited effort required. However, some tasks require specific personal characteristics or skills that not everyone has: for example, audio-recording tasks require native speakers.

More to the point, some micro-workers find these tasks perplexing, partly because requesters do not always clearly explain what their goal is. Diane could not make sense of why she was told to “draw a circle around a tomato” because “everyone knows what a tomato is, I hope”, and insisted that “I don’t know why”. Lack of understanding of the purposes of one’s own activity is known to entail psycho-

<footnote n="5">⁵ Interviewees’ names have been changed for confidentiality.</footnote>
social risks, against which micro-workers – who are formally not employees but independent contractors – are not protected.

Other psycho-social risks arise with content moderation tasks. Exposure to violence and other sensitive content can be distressing (Roberts, 2019), although our above-cited interviewee assures that she has found ways not to be personally affected. She admitted that she manages to resist because of fear that she might be given fewer tasks if she refused —while she badly needs the money.

Handling other people's personal data is also a source of psycho-social risks. Juliette, who listened to users' conversations with their virtual assistants (sub-section 3.1), felt uncomfortable at the thought that she might be unwittingly violating their privacy. She wondered whether they realized that human beings would listen to their “sexual delusions”, as she put it.

Another issue is isolation: micro-workers do tasks at home behind their screens and are rarely in contact with their peers, except a minority who join dedicated forums (less than 60% according to Berg et al. (2018)). In many cases, they also find it difficult to get in touch with platforms and clients. Another source of stress is fierce competition for tasks, due to high volatility of clients' demand: several interviewees said they keep their micro-working platforms always open or have set up alerts in order not to miss tasks when they are posted. This means unusual working times, including at night, for workers in low-income countries in Asia and Africa who do tasks for requesters mostly located in Europe and North America.

Against this background, it is easier to see why not all micro-workers engage fully with the activity, why some of them work (and earn) much more than others, and why micro-work is not a solution to the financial problems of all those who practice it. That said, what purposes does their activity serve?

# 4 How microwork feeds AI

It is now time to look at the other side of the market – the clients who post tasks on micro-working platforms and collect the outputs of workers’ effort. What do they look for, and why do they ask people to – to re-use the same example as above – draw circles around tomatoes in pictures of salads?

Leaving aside surveys which serve a range of diverse purposes (from marketing to scientific research), all the other tasks mentioned above have linkages to AI production, although this is not always immediately clear to workers.

Today’s AI production relies heavily on machine learning, a technique in-between computer science and statistics that “teaches” computers to find solutions from data, without each step having to be explicitly programmed (Alpaydin, 2014, 2016). “Learning” means that the quality of outputs can get progressively better over time, depending not only on the algorithm but also on the data given to it. Put differently, data are a necessary input just as algorithms. Micro-work prepares these data and supports the algorithms that handle it, as I show in what follows.
## 4.1 AI preparation

A first function of micro-work is what may be called “AI preparation” (Casilli et al., 2019) and comprises both generation of new data for machine learning, and “annotation” of already-existing data for better quality. For an example of data generation, take the above-mentioned task to read aloud short sentences and audio-recording them (sub-section 3.1). The requesters of such tasks are producers of virtual assistants who need to assemble large sets of diverse examples of voices, all pronouncing the same words, so that its vocal recognition algorithm can learn that they all mean the same despite differences in regional accents, tone, background noise, etc. It also needs examples of different ways to ask the same thing – such as, what the weather is like.

In turn, data annotation is needed for “supervised” machine learning algorithms, the best performing and most widely used ones so far. For example, to teach a computer to distinguish tomatoes from carrots, many images of these vegetables will not be enough, because the machine will not know what is what. Thus, the images should include tags that indicate, for each of them, whether it shows a tomato or a carrot, and ideally indications of where in the image the tomato or carrot is. After exposure to many images annotated in this way (so-called “training” datasets), the algorithm will be able to classify new, untagged images (“test” dataset) and determine whether they represent tomatoes, carrots or both. This is why micro-workers like Diane were asked to draw circles around tomatoes (sub-section 3.2). Their input will ultimately serve to build, for example, smart nutritional advisers that count calories based on the contents of a plate.

Many of the tasks described in sub-section 3.1 are for data annotation too. CV anonymization, with addition of tags to identify the location of sensitive information, was arguably meant to help some recruitment algorithm to understand the structure of a CV. Sorting documents such as videos or DVDs performs the same function, and so does content moderation – aiming to teach a computer to recognize violent, pornographic or otherwise sensitive content.

AI preparation can be partly automated, to the extent that algorithms can “learn” the data structure or its annotations from a first set of data prepared by micro-workers. However, there will always be a need for human input whenever new data are to be prepared. For example, selling a virtual assistant in a new country market will require a set of examples in that country’s language, dialect(s) and conditions.

## 4.2 AI verification

If AI preparation occurs at input level, to generate or enrich training datasets, another usage of micro-work is at output level, to review the results. This is AI verification. For example, checking the accuracy of speech recognition by virtual assistants is an example of verification, and so are checks of the quality of automated transcriptions of scanned invoices (sub-section 3.1). AI verification is more widely
applicable than preparation. First, it is needed with any type of machine learning al-
gorithms, not only with the supervised variant. Second, it is not only required when
AI solutions are being developed, but also, and more importantly, when they are al-
ready in use: for example, every time a company commercializes a virtual assistant
in a new country, it needs new, local workers to check speech recognition results in
the languages and dialects of that country.

4.3 AI impersonation

AI impersonation occurs when micro-work does not support the processes of data
production or algorithmic quality assurance, but replaces them when they fail. This
happens when humans outperform computers, either in terms of efficacy or cost.
The very idea that prompted Amazon to launch Mechanical Turk in 2006, was to
integrate humans directly into software programming when they were more efficient
than automating. According to (Irani, 2015, p. 225), this platform was "born out
of the failures of artificial intelligence to meet the needs of internet companies".
AI impersonation may seem unfair or even unlawful when use of low-paid humans
instead of algorithms is undisclosed – as in the case of the AI start-ups that outsource
work to providers in low-income countries rather than developing real AI, as one
interviewee Kader accused them to do (sub-section 3.1). This is sometimes just
a quick-and-dirty remedy to pressure from investors, but it is sometimes due to
genuine reasons: “faking” AI may be a way to understand the production process
of some service in the first place, before actually developing a suitable algorithm to
automate it.

These considerations hint that the need for micro-work is unlikely to be a tem-
porary one. As more industrial sectors integrate AI-based solutions, the need for
micro-workers to prepare datasets for AI and to perform quality checks on algo-
rithmic outputs will remain high. As long as wealth and income disparities across
countries maintain repositories of cheap human labor in the developing world, AI
impersonation will remain a cost-effective option.

5 Conclusions

Micro-work offers an opportunity for extra earnings to women with childcare duties who do not have a full-time job and do tasks primarily from home. It also attracts the inactive and more generally, lower-income (albeit highly-educated) persons. However, these extra earnings are often tiny or irregular, largely due to highly fluctuating demand from companies. The extent to which micro-work offers an opportunity for personal and/or professional development is doubtful, to the extent that tasks are mostly unqualified, that some micro-workers cannot give meaning to them, and that incentives to engage more intensely in it are scarce due to competition for tasks,
uneven working hours, and social isolation. While the study presented here focuses on one single country, France, its results are likely transferable to other settings.

And yet, micro-work is an indispensable ingredient of AI development. Consequently, current reflections about ethical AI should fully integrate the issue of labor standards in the backstage of automation. We should not look at AI as a pure product of techno-scientific progress: it also depends on economic cost considerations that lead companies to add humans to robots (or even to replace robots with humans) whenever convenient, and on socio-economic inequalities inherited from the past, whereby micro-taskers from low-income countries sometimes appear as a preferable alternative to highly-expensive engineers in the Western world. The consequences of AI on society are not independent of these hidden providers: for example, personal data leakages are all the more serious as independent sub-contractors potentially located anywhere have access to them, as recent scandals around voice-activated assistants demonstrate (Day et al., 2019).

Before even thinking of regulating this new labour force and strengthening its almost-inexistent social protection, it is necessary to at least recognize explicitly the contribution of workers to technological innovation, and the important role of this new form of work in the production and deployment of AI solutions.

# 6 Acknowledgements

This paper presents results of the “Digital Platform Labor” (DiPLab) project, co-funded by Maison des Sciences de l’Homme Paris-Saclay; Force Ouvrière, a workers’ union, as part of a grant from Institut de recherches économiques et sociales (IRES); and France Stratégie, a service of the French Prime Minister’s office. The platform FouleFactory/Wirk offered logistical support, and Inria provided complementary funding. For more information: http://diplab.eu/

I would like to thank current and former DiPLab team members, notably Maxime Besenval, Antonio A. Casilli, Odile Chagny, Marion Coville, Clément Le Ludec, Touhfat Mouhtare, Lise Mounier, Manisha Venkat and Elinor Wahal.

A preliminary version was presented at the “Automating the Crowd” workshop that took place at the Alan Turing Institute, London, on 22 January 2019, and I am grateful to organizers and participants for helpful feedback. Any remaining errors or inconsistencies are exclusively mine.

# References

* Alpaydin, E. (2014). *Introduction to Machine Learning* (3rd edition). MIT Press.
* Alpaydin, E. (2016). *Machine Learning: the New AI*. MIT Press.
Berg, J., Furrer, M., Harmon, E., Rani, U., and Silberman, M. (2018). Digital labour platforms and the future of work: Towards decent work in the online world. ILO Report.

Borgonovi, F., Centurelli, R., Dernis, H., Grundke, R., Horvát, P., Jamet, S., Keese, M., Liebender, A., Marcolin, L., Rosenfeld, D., and Squicciarini, M. (2018). Bridging the digital gender divide: Include, upskill, innnovate. OECD report.

Casilli, A., Tubaro, P., Le Ludec, C., Coville, M., Besenval, M., Mouhtare, T., and Wahal, E. (2019). Le micro-travail en France. Derrière l'automatisation, de nouvelles précarités au travail ? Report of the Digital Platform Labor (DiPLab) project.

Day, M., Turner, G., and Drozdiak, N. (2019). Amazon workers are listening to what you tell alexa. *Bloomberg*.

Difallah, D., Filatova, E., and Ipeirotis, P. (2018). Demographics and dynamics of Mechanical Turk workers. In *Proceedings of WSDM 2018: the Eleventh ACM International Conference on Web Search and Data Mining*, pages 135–143. ACM.

Forde, C., Stuart, M., Joyce, S., Oliver, L., Valizade, D., Alberti, G., Hardy, K., Trappmann, V., Umney, C., and Carson, C. (2017). The social protection of workers in the collaborative economy. Report for the committee for employment and social affairs, European Parliament.

Graham, M., Hjorth, I., and Lehdonvirta, V. (2017). Digital labour and development: impacts of global digital labour platforms and the gig economy on worker livelihoods. *Transfer: European Review of Labour and Research*, 23(2):135–162.

Gray, M. and Suri, S. (2017). The humans working behind the AI curtain. *Harvard Business Review*, (9):2–5.

Gray, M. and Suri, S. (2019). *Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass*. Houghton Mifflin Harcourt.

Hara, K., Adams, A., Milland, K., Savage, S., Callison-Burch, C., and Bigham, J. (2018). A data-driven analysis of workers’ earnings on Amazon Mechanical Turk. In *Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems*, CHI ’18, pages 449:1–449:14, New York, NY, USA. ACM.

Hirth, M., Hoßfeld, T., and Tran-Gia, P. (2011). Anatomy of a crowdsourcing platform - using the example of Microworkers.com. In *2011 Fifth International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing*, pages 322–329.

Ipeirotis, P. (2010). Analyzing the Amazon Mechanical Turk marketplace. *XRDS*, 17(2):16–21.

Irani, L. (2015). Difference and dependence among digital workers: The case of Amazon Mechanical Turk. *South Atlantic Quarterly*, 114.

Irani, L. (2016). The labor that makes AI "magic". AINow Summit, White House and NYU.

Kuek, S., Paradi-Guilford, C., Fayomi, T., Imaizumi, S., and Ipeirotis, P. (2015). The global opportunity in online outsourcing. Report, World Bank.

Posch, L., Bleier, A., Flöck, F., and Strohmaier, M. (2018). Characterizing the global crowd workforce: A cross-country comparison of crowdworker demographics. *CoRR*, abs/1812.05948.
Roberts, S. (2019). *Behind the Screen: Content Moderation in the Shadows of Social Media*. Yale University Press.

Ross, J., Irani, L., Silberman, M., Zaldivar, A., and Tomlinson, B. (2010). Who are the crowdworkers?: Shifting demographics in Mechanical Turk. In *CHI '10 Extended Abstracts on Human Factors in Computing Systems*, CHI EA '10, pages 2863–2872, New York, NY, USA. ACM.

Schmidt, C. (2017). Digital labour markets in the platform economy: Mapping the political challenges of crowd work and gig work. Report, Friedrich-Ebert-Stiftung.

Schmidt, F. (2019). Crowdproduktion von Trainingsdaten: Zur Rolle von Online-Arbeit beim Trainieren autonomer Fahrzeuge. Report, Hans-Böckler-Stiftung.

Tubaro, P. and Casilli, A. (2019). Micro-work, artificial intelligence and the automotive industry. *Journal of Industrial and Business Economics*, 46(3):333–345.

Wood, A., Graham, M., Lehdonvirta, V., and Hjorth, I. (2019). Networked but com-modified: The (dis)embeddedness of digital labour in the gig economy. *Sociology*, page (online first).