# Who Decides When Artificial Intelligence Works? Algorithmic Production as a Continuous Evaluation Process

Assia Wirth, Clément Le Ludec, Camille Girard-Chanudet, Maxime Cornet

<figure><img src="image_1.png" /></figure>

## To cite this version:

Assia Wirth, Clément Le Ludec, Camille Girard-Chanudet, Maxime Cornet. Who Decides When Artificial Intelligence Works? Algorithmic Production as a Continuous Evaluation Process. Digital Society, 2025, 4 (65), pp.1-26. 10.1007/s44206-025-00224-4. hal-05247876

HAL Id: hal-05247876

https://hal.science/hal-05247876v1

Submitted on 25 Sep 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>
<figure><img src="image_3.png" /></figure>

Check for
updates

# Who Decides When Artificial Intelligence Works?
## Algorithmic Production as a Continuous Evaluation Process

Assia Wirth<fnref n="1" />☐, Clément Le Ludec<fnref n="2" />, Camille Girard-Chanudet<fnref n="3" />☐, Maxime Cornet<fnref n="4" />

Received: 15 November 2024 / Accepted: 25 August 2025
© The Author(s) 2025

### Abstract

The intensifying deployment of AI systems in different spheres of our lives has raised concerns over their evaluation. The latter has often been discussed as a unified process that any model should undergo before its release and use. However, we argue that throughout their production AI systems are continuously assessed. A closer look at their supply chains reveal modes and practices of evaluation at each stage of their development. Based on empirical research of three sites of AI development, this article provides evidence for this continuous assessment. It points to the limits of common perspectives on AI evaluation and argues for a more grounded and comprehensive understanding of the ways in which AI systems can be and are assessed.

**Keywords** Evaluation · Artificial intelligence · Data work · Data annotation

<figure><img src="image_4.png" /></figure>

Assia Wirth
assia.wirth@universite-paris-saclay.fr

Camille Girard-Chanudet
camille.girard-chanudet@ehess.fr

<footnote n="1">¹ Université Paris-Saclay, Gif-sur-Yvette 91190, France</footnote>

<footnote n="2">² Université Paris Panthéon Assas, Paris 75005, France</footnote>

<footnote n="3">³ Centre d'Etude de l'Emploi et du Travail, Conservatoire National des Arts et Métiers, Paris 75003, France</footnote>

<footnote n="4">⁴ Télécom Paris, Palaiseau 91120, France</footnote>
# 1 Introduction

The development of artificial intelligence over the past decade and the initial deployment of “intelligent” systems have sparked numerous controversies and raised concerns about the consequences of their widespread adoption. Some authors predict massive job losses due to the automation of business processes (Frey, & Osborne, 2017), while others highlight concerns about biases embedded in these models (Angwin, Larson, Mattus, Kirchner, 2016), or raise alarms about the potential threats and human rights violations that the military and security use of these technologies could entail (Vold & Harris, 2023).

These concerns are exacerbated by the opacity surrounding the functioning of algorithmic systems, which often operate as “black boxes” (Pasquale, 2015) with outcomes that remain unexplained by their developers. In recent years, this opacity has become a growing issue as researchers point to the complexity and proprietary constraints of these systems, which hinder transparency and accountability (Burrell, 2016; Bechmann & Bowker, 2019).

Against this backdrop, increasing public attention has been drawn on the evaluation of AI systems. Algorithmic evaluation, audits, and ethical risk assessments are emerging as standard practices, driven in part by European Union (EU) regulations that set evaluation and validation requirements for so-called high-risk AI models. On the other hand, regulators do not define standard procedures for evaluating models and their outputs, nor do they give a stable definition of ‘evaluation’, limiting themselves to calling for human intervention to keep an eye on the action of the models. Current AI evaluation efforts can be divided in two groups.

The first addresses the issue of AI in terms of ‘risks’, sometimes ‘existential risks’ (Critch & Krueger, 2020; Hendrycks et al., 2023). These approaches are characterized by a commonexternal, non-technical, positioning towards algorithmic systems. They encompass scholarly work from disciplines such as ethics, social sciences, and political philosophy, which critically interrogate the implications of artificial intelligence for users, society, and the broader socio-political contexts of its deployment. This group includes authors discussing the notion of ‘safe AI’ (Burden, 2024), ‘AI values’ (Gabriel, 2020) or ‘ethical AI’ (Jobin et al., 2019). Impact assessment is often forward-looking, seeking to warn against the perceived risks of deploying systems in terms of weakening the job market, military uses of AI (Critch & Krueger, 2020), or even the automatic reproduction of discrimination (i.e model’s ‘biases’). This is done in terms of the potential impact of deploying models, and remains at a significant level of abstraction.

The second group is made up of authors who focus on the technical evaluation of models through the mobilisation and optimisation of indicators. Some indicators, which have become ubiquitous, are used directly in the model training phases to assess the impact of parameter variations - this is the case, for example, of the F-1 score, which summarises false positives and false negatives to assess the accuracy of predictive models (Derczynski et al. 2016). Others are specifically designed for use beyond the strict framework of the model, proposing indicators to estimate the validity of a model for a defined use case. These indicators are usually constructed by comparing a database annotated by a group of domain experts with the predictions

<figure><img src="image_5.png" /></figure>
proposed by an algorithm downstream of the training procedure. An extension of
this approach is to perform an a posteriori critical audit of the models by evaluating
the model's biases in terms of a pre-classification of the input data compared to the
proposed outputs. For instance, Buolamwini and Gebru (2018) uncovered significant
misclassification, particularly for women of colour, highlighting the biases that can
be embedded in AI systems.

This division into two approaches to model evaluation is regularly found in the
scientific literature, both technical and relating to regulation and public policy. John
Burden, for example, describes two approaches to evaluation, performance-oriented
(evaluated via the optimisation of an indicator) and capability-oriented, involving a
more holistic assessment of the model's performance on the 'multiple task instances'
that make up the model's action space (Burden, 2024). Similarly, Mélanie Gornet and
her colleagues speak of a tension between actors advocating for a strictly technical
definition based on the optimisation of a set of standard indicators (academia) and
those favouring a more flexible, interpretative approach (industry) (Gornet, & Her-
man, 2024).

Therefore, the question of how AI should be evaluated remains a topic of ongoing
debate, as it is difficult to define a single, universally applicable methodology (Floridi
& Straits, 2020; Krijger, 2024). In particular, some authors point to the shortcom-
ings of technicist approaches based on indicators, highlighting the fact that the inclu-
sion of expert knowledge (which requires the application of contextualised practical
knowledge) in training databases (which are by nature cut off from the context of
action) is not self-evident, leading to a potential mismatch between the representa-
tion of the problems to be solved contained in the data and the use of the models in
production (Lebovitz et al., 2021). Burden (2024) uses the example of OpenAI and
their Chat-GPT service to illustrate how performance indicators can legitimise the
commercial use of a model, even though a risk in terms of ‘capability’ (in this case
the risk of disinformation and the spread of false information) is known and identified
by the company deploying it. By doing so, they advocate for a broader evaluation of
AI models that encompass the “risk” of AIs, but without providing a clear roadmap
regarding the steps required for this kind of evaluation.

In this article, we analyse choices and strategies mobilized by actors along the
model's production chains, by looking at evaluation tools and processes along the
production process. Ultimately, we argue that emphasizing the human dimension of
data work complements purely technical perspectives on AI models, offering a more
comprehensive understanding of their functioning.

## 1.1 Understanding the Human Labor behind AI and its Challenges for Evaluation

Discussions about the evaluation of algorithmic systems tend to focus narrowly on
assessing the models themselves, while early empirical research in the social sciences
reveals that AI systems are far more than just technical components and rely heav-
ily on large-scale human data labor usually made invisible by design (Seaver, 2017;
Christin, 2020; Gray & Suri, 2019). The latter developed across different types of
organizing structures, ranging from delocalized models such as gig work platforms
(Irani, 2015) or BPOs (Chandhiramowuli et al., 2023), to in-house out-sourcing
(Girard-Chanudet, 2023a, b). From selecting training datasets and annotating data to calibrating models and correcting outputs, AI systems are shaped through extensive production chains involving iterative exchanges, negotiations, and feedback among professionals with diverse expertise (Le Ludec et al., 2023). For instance, in autonomous vehicle development, humans annotate images to train machine learning systems to identify pedestrians and road elements (Tubaro, 2019; Schmidt, 2019). Tubaro (2020) identify three key human roles in AI production: “preparation” (data labeling), “verification” (quality control), and “impersonation” (simulating AI tasks). While often cited as key sites of production of biases exhibited by AI models, Miceli et al. (2022) argue this data work should rather be evaluated through “a power-aware perspective” (1), which sheds light on its historical, material and epistemic realities. Thus, studying digital labor practices is here seen as a form of AI evaluation in and of itself.

This article introduces a novel perspective by bridging digital labor studies with valuation studies (Muniesa, 2003; Vatin, 2013). The latter deconstructs the notion of objectivity in evaluations, viewing valuation as a product of social constructions. It examines the plurality of tools, frameworks, and cultural, moral, and political assumptions that underpin these activities, as well as the diverse outcomes they can produce (Helgesson and Muniesa, 2013). By integrating these two fields, this approach reveals that AI evaluation is anything but monolithic—it is diverse, encompassing a wide range of evaluators, evaluation objects, and quality criteria. By focusing on the often-invisible actors behind the scenes of AI development, this approach highlights how these individuals are engaged in constant evaluation and problematization of the systems' components.

Science and technology studies have long argued that knowledge cannot be isolated from the practices and contexts in which it is mobilised (Knorr-Cetina, 1999). In deploying knowledge in action, actors mobilise a body of ‘tacit knowledge’ in their practices, forming an unspoken body of knowledge that complements ‘explicit knowledge’ and makes it mobilisable (Polanyi, 1958). In the creation of databases, this means that the actors responsible for formalising the taxonomies that make up the structure of the datasets seek to inscribe their situated practices within quantifiable metrics. For Bechman and Bowker (2019), it is this displacement, where a subjective way of seeing the world is inscribed within a system that aims to objectify and extend it, that produces exclusion and discrimination. However, we know that the design of these categories takes place in a negotiated relation of production, extended to all players involved in the process and structured by power and hierarchical relations (Le Ludec et al., 2023).

Our approach examines how knowledge embedded in AI systems reflects the situated perspectives of these actors at every stage of production, becoming visible in their evaluation and problematization activities (Jaton, 2021; Daston & Galison, 2010). Based on this perspective, the article argues that every stage of AI production serves as a site for evaluation and adjustment, with these overlapping practices collectively shaping the final product. However, the invisibilization of labor leads to the erasure of these evaluation activities in public and political narratives about AI. To address this, we advocate for an evaluation framework that encompasses the entire AI production process. Moving beyond technical audits conducted at the end of develop-

<figure><img src="image_6.png" /></figure>
ment, we call for a comprehensive evaluation approach spanning all stages of AI production cycle. Building on extensive empirical research of various AI supply chains, our proposed approach to AI evaluation make central the grounded practices that have up until now remained imperceptible to more technical-based evaluation frameworks. This bottom-up perspective allows us to simultaneously expand and refocus current understanding of AI evaluation and by so doing it contributes to scholarship advocating for more representative hence distributed recognition of work undertaken by various actors of AI supply chains.

## 1.2 Methodology: a Cross-Analysis of Three Sites of AI Work

This research leverages a substantial body of data collected through fieldwork in various AI production settings. The research focuses on three key sites (Table 1) investigated separately by the four co-authors. Across all cases, semi-structured interviews were conducted with project stakeholders, including project managers, data scientists, annotators, and annotation software developers. In addition, a variety of documentary materials were collected and analyzed, encompassing work documents, annotation guides, and communication materials.

The first site explores the development of facial analysis technology. It examines the outsourced production of datasets and models on freelance platforms, which enable clients (often located in the global north) and workers (often located in the majority world) to connect and undertake entire data projects. It is grounded on interviews with 37 data annotators based in Bangladesh (4), Georgia (1), Ghana (1), India (3), Indonesia (1), Kenya (14), Nigeria (3), Pakistan (4), Tunisia (1), Uganda (3) and Venezuela (1) and interviews with 10 ML professionals working in commercial or public research capacities in France, Spain or the UK. The second site examines externalisation chains between French AI startups and Malagasy Business Process Outsourcing (BPO) organizations. The models developed cover a wide range of use cases (defence, manufacturing, administrative support, etc.) and technologies (NLP, computer vision, etc.). The fieldwork includes interviews with workers from 11 startups (30), data annotators (147) and executives (13) from 7 BPO companies, and makes it possible to describe -- at least partially -- the production processes of 158 models. The third field site follows the development of an AI-based anonymization tool for judicial decisions by the French Supreme Court. This tool is designed to

<table><caption>**Table 1** Fieldwork summary</caption><thead><tr><td></td><td>Fieldwork sites</td><td>Actors encountered</td><td>Supply chain stages</td><td>Fieldwork dates</td></tr></thead><tbody><tr><td>Field one</td><td>Online freelance platforms and ML commercial and research professionals</td><td>Data annotators, quality analysts, data management</td><td>Training sets production and procurement</td><td>2022–2023</td></tr><tr><td>Field two</td><td>Data annotation subcontractors in Madagascar and Cameroon</td><td>Data annotators, Quality analysts, Data management</td><td>Training sets production and procurement</td><td>2021–2022</td></tr><tr><td>Field three</td><td>French Supreme Court</td><td>Data annotators, Data scientists, Judicial management (i.e. judges)</td><td>Training set production, model training, model surveillance</td><td>2021</td></tr></tbody></table>
automatically obscure identifying information from legal decisions for open data dis-
semination. Within the Supreme Court, a team of judges, data scientists, developers,
and approximately twenty annotators collaborate on training, maintaining, and moni-
toring the AI algorithm responsible for this anonymization process.

The originality of this research lies in its cross-sectional approach to investigating
varied AI design frameworks, comparing tools with different goals (facial recogni-
tion, automated payment systems in cafeterias, or judicial decision anonymization),
host organization types (start-ups vs. public institutions), and organizational struc-
tures of design work (ranging from fully internalized to partially outsourced and
offshored models). This cross-analysis of diverse cases reveals recurrent processual
dynamics at a granular level, offering critical insights into the broader practices and
challenges of AI design. Notably, across all cases, the pervasive presence of micro-
evaluation processes becomes evident, permeating each stage of the AI production
chain and gradually solidifying the algorithmic structure. Tracing the making of an
AI project—understanding its initial motivations, the actors involved, and the pro-
cess of problem definition—offers valuable insights into the networks of power that
shape AI development.

In the first section, we explore the objective-setting phase, highlighting how proj-
ect goals are shaped by the organizational context in which models are developed.
Clear and well-framed objectives are essential for AI projects to succeed; without
them, algorithms lack the precise scope needed to perform tasks meaningfully. In this
section, we tackle the overlooked challenge of goal evaluation, a critical aspect fre-
quently neglected in AI production contexts. We then turn to the development phase,
focusing on annotation as a pivotal evaluative process. Far from being a purely techni-
cal task, annotation involves annotators’ perspectives, which filter and reinterpret the
objectives initially established. At this stage, we show that evaluation mainly focuses
on the data and the categorization systems used to guide their annotation. Finally,
we examine the model’s final evaluation phase, that occur during its encounter with
real-world conditions. This phase asks whether the model can accurately capture and
respond to practical realities, and whether its outputs align with the intended purpose.
It is only here that the algorithm’s ability to function reliably in operational contexts
is ultimately tested.

## 2 Evaluating the World to Turn it into Data

### 2.1 AI Begins with a Need: Goal-Setting

AI projects are fundamentally designed to automate the processing of information,
whether it involves images of animals, medical reports, or audio recordings in urban
environments. This broad applicability means that AI projects can be driven by vari-
ous motivations. Some projects are launched with a clear commercial interest, such
as using AI to streamline labor-intensive processes, such as cafeterias using image
recognition for automatic checkout systems. AI projects can also be launched as a
response to external legal or regulatory requirements, as in the case of the French
Supreme Court, which developed an algorithm to automatically anonymize judicial

<figure><img src="image_7.png" /></figure>
decisions in response to the obligations set by the General Data Protection Regulation (GDPR).

Initial objectives serve as a guiding framework, which conditions the structure of the training dataset, the design of the model, and the metrics used to evaluate its success. These different configurations have significant implications for the trajectory and subsequent functioning of AI systems, framing directly the outputs of the algorithm. The case of the French Supreme Court's anonymization project (Field 3) exemplifies this dynamic:

> « For the definition of categories, as with everything else, it's a collective process—it's a movement. There's a saying: "Law is born from fact." That's exactly it. Like much of our profession, it's a constant back-and-forth between fact and principle. We try to identify the principles that best fit the facts, and when we encounter a new fact, we strive to align the rules with it as effectively as possible. It's a continuous exchange: "What happens if we apply this rule?"—whether in a theoretical sense, as we do here, or in practice, as in litigation. It's really about going back and forth between the objectives we know, the realities we uncover along the way, and this ongoing reassessment of risk. »

> — Judge working on the AI project

Here, the key decisions about the project's objectives and the categories of information to be anonymized were made by groups of experienced judges, chosen for their expertise in specific areas of the law. These magistrates met behind closed doors to determine the criteria that would guide the AI's development, thereby exercising significant influence over the project's direction. The selection of categories for decision anonymization was primarily driven by the legal expertise of the individuals involved, who developed the taxonomy by aligning it with relevant legal frameworks (GDPR, open data law). It appears clearly how their social position influenced the categorization choices used in machine learning.

In a commercial context, final clients often work with AI companies to develop models and plays a crucial role in shaping the AI solution from the design phase onward. Interviews (Field 2) reveal that developing an AI model typically begins with a collaborative phase between AI companies and their clients, involving meetings to define the model's functionality. During this phase, clients' feedback often guides the development process providing a domain-expertise to AI companies. For instance, in a project involving a call-bot meant to replace call center workers, these meetings helped both parties anticipate user interactions with the call-bot before model's training by providing examples of questions and answers. This focus is sometimes evident through the presence of employees specifically tasked with integrating expert feedback into the AI models and annotation guidelines. This ensures that the solution aligns closely with the customer's needs and expectations.

This initial step can take place in various forms, each shaping its outcomes differently. Objectives may be defined collectively by a group of stakeholders or individually, and their relevance can be assessed through preliminary evaluations, or they may be implemented without such scrutiny. This process of 'problematization' translates
broad goals into specific, actionable objectives across different organizational levels, from high-level decision-makers to project managers, or technical staff. that culminates in the first version of an annotation guide, which formalizes the project's goals into clear instructions for data annotators (see: 3.1). The problematization phase of AI work is a highly normative process which leads both to framing the scope of the training dataset, and to the creation of a robust classification system, which the AI model will later replicate. Thus, defining the project's objectives is inherently an exercise in power—it dictates what becomes visible or invisible within the system, deciding which aspects of reality will be highlighted and which will be ignored (Bowker & Star, 1999), reflecting the values and priorities of those involved in the decision-making process. This configuration highlights the political dimension of setting AI objectives, as it shows how control over this process is closely guarded and involves negotiations about what should be prioritized (Miceli et al., 2020; Miceli, Posada and YanMiceli et al., 2022; Girard-Chanudet, 2023a, b). The expert knowledge and decision-making authority of AI models' developers and requesters, illustrating how the definition of objectives is not just a technical step but a deeply political act, reflecting broader professional values and power structures.

## 2.2 Organizing Work: the Construction of Datasets

> *Beyond the type of producer and the dataset mobilised, the data flow itself varies greatly according to the production context. Our research identified several types of AI start-ups and projects, each with distinct approaches based on their objectives and resources. Some focus exclusively on developing a single AI product, others, like “AI agencies,” develop solutions based on a particular model category (e.g., computer vision) and adapt them across various projects.*

AI companies sometimes start development with open access datasets, but in many cases the development reveals the limitations of open datasets. For instance, in field 2, a company executive interviewed regarding the development of a call-bot told us that they quickly realized that for their AI assisting tool to respond with high specificity, particularly in recognizing brand names, open datasets were insufficient, highlighting the need for tailored data to meet customer demands.

In academic research (Field 1), data is either collected during research projects, especially when projects' aim is not to develop a model but to improve data processing, or comes from “open”, pre-annotated datasets, typically when the goal of the project is the model itself. Researchers interviewed explain how open access datasets are often sufficient to test out and undertake initial training of new models. Public administrations (Field 3), on the other hand, usually work with their own data.

Regardless of the production set-up, the involved actors must determine key data characteristics and, in particular, on the organisation of data flows. How and in what form will the data be sent to the annotators? How often? All these decisions give rise to negotiations between the actors in charge of annotation, those in charge of model production and those in charge of deployment. Frictions can arise, particularly between model producers and their clients, over the ownership and reuse of datasets.

Model producers, particularly AI agencies, face a major dilemma, which is usually settled by contract at the beginning of the commercial relationship. In fact, train-

<figure><img src="image_8.png" /></figure>
ing and maintaining a model for a single client is not very advantageous, so these companies aim to develop reusable “off-the-shelf” models. For instance, one company interviewed trained a model for sorting recyclable waste used on a sorting line could be adapted for use across the entire industry with a few minor adaptations. The question of ownership of the data used to drive the original model then becomes a problem. Indeed, it is often not very strategic for clients — who spend a considerable amount of time co-defining the model and adapting the proposed outputs to their own ‘business’ use cases — to allow their competitors access to a ready-made model.

What’s more, as we’ll see later in this article, these models are likely to contain a wealth of knowledge and representations of the customer’s production processes, which can pose major problems in terms of industrial secrecy. Thus, the framework for re-use of both datasets and models may be the subject of acrimonious negotiations. These considerations come on top of important questions about the legal framework for the use, circulation and processing of personal data.

## 2.3 Evaluating Subcontractors’ Depending on Clients’ Perceived Data Quality Needs

These issues around data and its use lead to wide variations in the implementation of data processing chains. Indeed, the ways in which data work is organised are extremely diverse. While the literature has focused on digital labour platforms where ‘micro-work’ is exchanged, such as the American platform Amazon Mechanical Turk, there are many other forms of outsourcing these activities (Muldoon et al., 2024).

Outsourcing arrangements vary in how they integrate workers into the AI model production process, significantly impacting their working conditions. On micro-work platforms, independent workers perform short, low-paid tasks, and have generally very limited interactions with final clients. General freelance platforms, allow more skilled workers to negotiate both their rates and expected volume directly with the client generally discussing on a day-to-day basis with clients. Closed platforms, such as Remotask or Appen, mediate the process: clients negotiate with them and then allocates the work to a pool of annotators selected, trained and managed by the platform. Business Process Outsourcing (BPO) companies handle a range of business support tasks, alongside data entry and customer support, content moderation and data annotation. Unlike platforms, BPO workers are generally salaried, and have regular meetings with final clients, especially for AI projects. Finally, some data processing is internalised within organizations, especially when data is considered particularly sensitive, or in the case of start-ups that have not yet structured their processing chain.

The choice of service provider depends on the sensitivity of data. Potential data breaches are easier to control when outsourcing to a BPO under contract and subject to audit, compared to dispersed, independent platform workers. Similarly, the type of organisation also hinges on the expected quality and complexity of annotation. A closer relationship between data scientists and annotators better align will the model’s needs and the evolving categories of the annotation guide encouraging organisations to foster regular interactions between final clients and workers i.e. general freelancer
platforms, closed-platforms and BPO. In contrast, the annotation produced on large open platforms are often considered low-quality by AI companies. The latter often have no way of asking questions, and little incentive to do so given the dispersion of tasks. In-house annotation, though more expensive, is typically regarded as higher quality.

Honestly, we looked into Amazon and its system, but we quickly abandoned it. From an annotation standpoint, the quality is terrible. We wanted people to develop a certain level of expertise. If randomly chosen individuals do this for a small amount of money, the quality [of the work] will most likely suffer. If 100% of your annotations are flawed, it's completely useless.

— Chief Technical Officer, AI Company

In this appear the first tension between contradictory valuation practices depending on actors' objectives. Here, data scientists often value form of labour organisation that allow annotators to develop « skills » — understood as a set of untold work-related practices that have been codefined over long period of time by annotators and data-scientists — and workers engagement in the project, mainly to avoid what they perceive as forms of data sabotage. Platform workers on their side value task that they perceive requiring a lower time-investment on their part, with better pay, and « safe » clients that won't try to withhold payment. One way to resolve this tension is to stabilize the work-relationship, either through the use of formal employment contracts, allowing to attach annotators to specific projects, or through service contracts allowing re-occurring interactions between annotators and clients.

The choice is also influenced by the volume of data to be processed. Large industrial projects with very large volumes of data to be processed in parallel - such as autonomous car projects - require a very large number of annotators, which small BPO companies have difficulty providing. The choice then generally falls either on the few BPOs with international reach that can quickly provide more than a thousand positions for a project, or on the large, closed platforms that are capable of extremely fast and flexible volume adjustments while maintaining high quality. Overall, this discussion raises controversies around data quality and its impact on AI models. A recent controversy over the use of the Amazon Mechanical Turk platform suggests that data quality is declining, affecting the reliability of psychology and computer science studies (Chmielewski, 2020; Webb, 2022). Moreover, recent research indicates that “33–46% of crowd workers used LLMs when completing the task”, further calling into question the validity of data collected through this platform (Veselovsky, Horta Ribeiro, and West, 2023).

Access to high-quality human data, thus, constitutes a critical evaluation phase for the quality of AI models, and is considered as such by AI developers who spend times to choose the most effective solution in terms of costs and quantity. Ultimately, AI developers apply a form of evaluation that can be summarised as follows: what form of labor organization will yield the « best » data (Laux et al., 2023)? This dimension has been further highlighted by recent research showing that “professional annotation companies outperform crowdsourcing” (Rädsch et al., 2024). Comparing BPO
and crowdsourcing models, they found that the latter’s workers were more prone to errors (around 20% vs. 44,8% of images with severe errors) and spam (0% vs. 20,1% of spam images).

# 3 Evaluating Data to Turn it into Training Sets: The Centrality of Annotation

## 3.1 Drafting Instructions: The First Annotation Guide as a Tool for Testing the Organisation of Annotation

In most outsourcing cases, once a contact has been established between data annotator(s) and clients, a first step is therefore to negotiate a contract. To that end, the clarification of the annotation work required by the potential project is key. In many cases, the clients’ original guidelines and scope require fine-tuning, or the dataset still requires some assembling or formatting. It is then up to the data annotators to request the information necessary to adequately assess the size and difficulty (and thus cost) of the project.

Based on this information, data annotators can assess the feasibility of the project estimating necessary number of hours and workers needed. In the case of BPO, this initial phase often includes the test of the project. Sometimes handled by the Research & Development (R&D) unit, this test involves revisiting the client’s instructions and applying them to an initial batch of data. The goal is to identify potential issues or inconsistencies in the guidelines before scaling the work.

Frequently, the client’s original guidelines and scope require adjustments, or the dataset may still need assembling or formatting. By conducting these tests early, the BPO can anticipate and resolve problems, refining both the instructions and processes to better align with the client’s expectations and avoid issues during the broader data annotation phase as well as refining the pricing.

[Interviewer: ] So they didn’t really think about the potential categories that might exist...?

[Interviewee: ] Yeah, or about the edge cases (...). So we make suggestions when asked, yes, but when we see that the edge cases aren’t being addressed, we put a stop to it. We tell them, ‘We can’t handle this particular case if you don’t provide instructions’.

— Director, BPO Madagascar

Drafting annotation instructions warrants its specific set of skills and know-how. It entails having a good grasp on the objectives and requirements of the dataset as well as a realistic understanding of the practicalities of data annotation techniques. In most cases, an effective instruction manual is the result of a constructive exchange between the data annotator(s) and their clients.
If not properly negotiated, unrealistic objectives or instructions are prone to lead to underperforming dataset and thus client frustration. As long as the latter is deemed unusable, clients may ask for reworks which adds extra work for both parties. When annotation does not align with the clients' expectations, blame will most often fall onto the data annotators, regardless of the actual cause of this mismatch. For instance, an interviewed freelancer explained how some clients may decide that the delivered work is not up to their standard and fail to validate the transaction through the platform. Even though in one case, the freelancer contested this outcome and lodged a complaint through the platform, they were successful in getting their due. This underscores the power dynamics which structure these relationships and often reinforced by the organization of the supply chains.

In fact, client retention will push data work management to take the blame. These dynamics are further exacerbated in platform-based work, where the system is designed to favour clients during disputes, enabling them to retain workers' wages until they claim to be satisfied. The power imbalance is further bolstered by the commonly held misconception that data annotators are unskilled and therefore the most likely to make mistakes.

However, a closer look at the realities of this crucial step of dataset construction tells a different story. A key aspect of being successful as a data annotator lies in being effective and realistic when initially assessing a project. The downfall of this is that in some cases data annotators such as freelancers end up providing consulting services on project definition and planning free of charge, sometimes without even securing the annotation contract in the end.

> That was my first project freelancing. (...) So, some clients, they take advantage of the fact that, oh, this person is new (...) So, the way the client said it was like, “it’s just adjustments, it’s annotated, there are just some bounding boxes just to remove or adjust. So, it sounded it could be easy. So I started it. It was my first project. I don’t want a bad review. The budget is so low. I just had to finish it up. It took months and I was the only one working on it. The budget was really low… I think, $60 or so.

> — Data annotator, Nigerian freelancer

These types of predatory and exploitative experiences are often found among freelance platform workers' early experiences. Despite their technical skills, they may lack the project management skills and expertise required to viably assess their client's needs, who often take advantage of that. However, such abuse are usually overcome through data annotators' experience and know-how.

What constitutes « good » guidelines and the criteria on which they are evaluated can therefore vary heavily between the client and the annotators. Annotators value clear and actionable documentation, that minimize uncertainty surrounding client's expectation. Even if not always formalized, the first discussion around the framework under which the annotation job will be conducted also constitutes a negotiation regarding whom will have to undertake the formalization of the coordination resources constituted by the guidelines. A tension in the valuation process arise.
Some clients leave the task to annotators, who often have to undertake unpaid work before beginning the annotation job. The negotiation of what « good » guidelines constitutes a starting point of an evaluation process between the actors at stake on data labeling production.

Quality control and renegotiation. Are the specifications, work organisation and data sufficient to solve the problem?

After the first series of iterations, which serve as an initial test of the classes defined with the customer, the production chain can be set up. At this stage, depending on the type of relationship, a more or less strict hierarchical chain is established. Within BPOs, a team of annotators is recruited and managed by a team leader, sometimes including the annotators who carried out the test phase. The team performs the tasks (*annotation phase*), which go through successive stages of quality control before being delivered (*verification phase*). On freelance platforms, these control stages are the result of direct bilateral negotiations between the client and the worker.

The daily confrontation with the data, then reveals a series of problems, both related to the definition of the classes and to the production process itself (inadequacy of the tools, etc.), which were not foreseen during the testing phase. A series of renegotiations and redefinitions is then necessary, usually throughout the production phase. The consensus on class definitions necessary for production is inherently temporary and shaped by feasibility. All actors, from customers to annotators, work to agree on the meaning of categories in a classification system. The challenge is to balance a fine-grained categorization with categories broad enough for annotators to understand and apply, despite working with raw materials, which in itself offers a biased view of reality. For example, it's almost impossible to accurately define the degree of rust on a utility pole based on an image from a city surveillance camera, which is sometimes distant and blurry. The whole process is then open to negotiation, from the capture device, to the training process and the annotation tools.

> It's not that it wasn't right at all, it's just that there were a lot of things missing from the specs. When you see the images, you can see that there are a lot of categories that aren't in the specs, for example. And then we pass them on to the customers, and they add them to their internal documents, after validation by the internal people there. (...) [For example, for a project to] annotate waste, (...) from the outset we couldn't really delimit what would be in it or what wouldn't be in it. and there were a lot of things missing in fact (...). Because we really had to separate the transparent objects, the opaque objects and the tinted objects, the papers, the plastic bags, the stretch bags, the non-stretch bags ... ! there were quite a few things missing.

> — Project Leader, Malagasy BPO

The production chain, which unfolds a series of temporary compromises that enable action, is thus quite vertical. It is characterised by a series of negotiated compromises regarding taxonomy, object classes, data capture devices, 'raw' data and the technical tools used all of which based on perceived needs. The extent of changes made to the guidelines from the test phase varies greatly depending on the project.
Uncertainty of input data and classes to be predicted as an indicator of model adaptation needs.

The combination of uncertainty in the data and aforementioned negotiated adjustments lead to variations in the extent of guideline changes from the test phase, depending on the project. As a result, major adaptation phases can take varying amounts of time, depending less on the technical complexity of the model and more on expected standardisation of the outcome. In cases with consistent input data and clear class boundaries, adaptation may be brief, while in cases with more fluid classes, it may continue indefinitely.

For example, a French entrepreneur who outsources the production of his datasets to a Malagasy BPO describes how one of the computer-aided vision models produced by his company - used to check the quality of the organisation of the cables inside fibre optic cabinets - works almost autonomously, without any new annotation phases, while another model produced by his teams - enabling the recognition of food on trays in the company restaurant - has required a constant flow of annotations since its launch and has never really left the test phase in five years of production.

The complexity of the two models and their supporting software components are quite similar here, differing mainly in the expected standardisation of the input data and the sharpness of the class boundaries that the model is asked to predict. It appears that it is indeed very difficult to predict the detailed contents of a plate from a simple photograph, especially when the elements of the image - crockery, other items on the tray such as telephones, keys or canteen cards, but also the layout of the dishes themselves - are highly variable.

Similarly, in cases where the use of the model is not perceived as very critical - and, consequently, where the acceptable error rate is relatively high, as may be the case for certain more playful consumer applications - the adaptability of the model, which allows it to stick as closely as possible to the role assigned to it and to follow its evolution, is perceived as less important.

The training dataset is thus never truly final, but a temporary point of consensus that materially inscribes a shared representation of the phenomenon being modelled. In other words, the data are never “clean” - to use the terminology of statistical practitioners - but adapted or not to a particular use, situated both in time and in the work-relationship. This dynamic, constantly renegotiated and distributed throughout dataset production calls for the implementation of fine-tuned, constantly updated levels of evaluation and control to ensure the consistency of the dataset.

## 3.2 Organizing Annotation Production: Digital Infrastructure Shaping the Process

In all fieldworks, we observed one key method of controlling annotation is through organizing workflows and establishing communication channels that enable continuous feedback and adjustments. These channels include regular debriefing meetings between annotators and supervisors, on-demand check-ins, and digital chat platforms for real-time problem solving. In some BPO companies, interactions with clients through messages are frequent, with meetings sometimes organized on a weekly basis, especially during the test phases. Most successful freelance projects are sup-

<figure><img src="image_9.png" /></figure>
ported by regular use of the platforms’ instant messaging function and regular online
calls between the worker and their client.

The annotators let me know about any issues or questions they come across
right away

— Annotation team manager, French supreme court of justice

Through these interactions, a collective expertise develops, moving beyond individ-
ual practices to create a shared knowledge base. This collective knowledge is crucial
for ensuring consistency and quality in annotation practices, leading to more reliable
and homogeneous training datasets.

Another approach to control annotation involves the digital interfaces provided to annotators, which play a critical role in shaping how the tasks are performed (Fig. 1). They restrict the choices available to annotators, for instance by presenting a limited list of labels, requiring annotators to prioritize certain documents, or include performance metrics to track productivity. In some cases, annotation interfaces even incorporate elements of gamification such as points-based system, rewards and daily leaderboard ranking the top performers, adding a competitive element to the task. These digital tools enable continuous managerial oversight over annotation tasks: metrics such as the number of annotations completed, the speed of work, and error rates can be tracked in real-time.

<figure><img src="image_10.png" /><figcaption>**Fig. 1** Graphical representation of an annotation interface. The image shows a photo of a canteen tray with a number of items on it, with the various labels attached to the items on the left: ‘plate’, ‘small fruit’, ‘free bread’, ‘traditional beef stew’. © Aurore Le Ludec (Le Ludec, 2024)</figcaption></figure>
## 3.3 Unequal Relationships Between AI Designers and Annotators Shaping Key Decisions for AI Models

This negotiated aspect of the dataset should not, however, obscure the power inequal-
ities that exist throughout the chain. The client retains most of the decision-making
power in the negotiation phase and can decide to withdraw from the relationship if the
trial phase fails. As we have seen, the importance of negotiating specifications during
the test phase is paramount for annotators, as it sets production expectations through
compromise. While annotation companies and freelancers can use this expertise to
help define realistic specifications, they are also held responsible for any failure and
bear most of the costs of a client's withdrawal. The time spent in the testing phase is
then not recouped when the project moves into production.

This unequal relationship permeates the entire production chain and largely deter-
mines how work is organised. Across settings, workers systematically adapt to what
they perceive to be the customer's expectations at a given moment, rather than to a
faithful representation of the phenomenon they have to deal with. This positioning
sometimes leads them to apply classes that they perceive as illogical, but that are
more in line with the customer's representation of the phenomenon.

> Sometimes we say to ourselves that such and such a video shouldn't appear, and yet the client says it should. Sometimes there's a disagreement between us and the client. Everyone has their own vision. But it's always the client who has the final say. To do this, we use a question file where we put the videos about which we have doubts. The customer will see whether a particular video is better characterised by him or her. If we have any doubts, we put them in this file along with our opinion and we ask the customer for their opinion. They look to see if our opinion of the video reflects what they want. And they give their opinion, whether it's OK or not.

> — Malagasy annotator

The extent to which annotation tasks are framed and controlled depends on the anno-
tators' hierarchical position within the organization. Annotators who hold privileged
positions, such as data scientists or domain experts like judges or physicians, can
be afforded greater flexibility in how they execute their tasks. Their expertise grants
them a level of autonomy that allows them to adapt and interpret the annotation
guidelines with a certain degree of latitude. In contrast, annotators in lower hierar-
chical positions, who represent the majority of the annotation workforce, typically
face much more rigid control mechanisms and close monitoring, with little room for
deviation from prescribed methods.

However, in some cases, the relative invisibility of annotation tasks, conducted at
a distance from strategic decision-making centers and perceived as a low-skill and
mechanical task, can paradoxically result in less scrutiny, granting annotators more
leeway in how they approach their work. These organizational configurations, which
determine the level of control or flexibility granted to annotators, have significant

<figure><img src="image_11.png" /></figure>
implications for the structure and quality of AI training datasets, and, in turn, for the
overall functioning of the models.

Here, competing definitions of what should and should not appear in the dataset
crystallize tensions beyond the product itself within the production chain. In addition
to assessing customers' expectations of the model, annotators must also evaluate the
degree of autonomy they can enjoy within the chain. In addition to their task, they
evaluate the structure of the production chain, with the aim of inferring their position
within it and what is expected of them.

This anticipation has an effect on the data set. The degree of independence given to annotators can backfire. When they have a high degree of autonomy in the way they carry out annotation, they also take on a large part of the work of coordinating the action - the dataset must, come what may, be in line with the uses the model designer wants to make of it. Their strategies are part of this tension: they must avoid initiatives that would cause the model to drift away from the designer's representation of it, while at the same time avoiding over-soliciting the designer, and for freelancers, guaranteeing a form of added value in the service provided. In this way, annotators' evaluation strategies are based more on the degree of autonomy they enjoy in their decisions than on explicit class-related claims, the model being perfectly identified as ultimately belonging to the client.

# 4 Closing the Loop: Evaluating the Algorithm in Relation to Data and Thus to the World

## 4.1 The Dataset as a Key Tool for Evaluation

Once an algorithm has been trained, the validity of its output needs to be assessed
on a large scale to ensure that the model is working as intended. Models can then
be distinguished in terms of the consequences of their real-world application, the
complexity of the product or service being automated, the level of external scrutiny
faced by the organization, and the sensitivity of the information used. These factors
determine not only how performance metrics are calculated, but also if workers are
kept in or out of the loop.

In highly sensitive contexts where the cost of errors is significant, such as in medi-
cal diagnoses or automated video surveillance, human workers can act as algorithmic
supervisors: they manually review the outputs generated by the algorithm on a case-
by-case basis, checking each result and correcting errors as they arise. However, this
approach is time-consuming, labor-intensive, and not scalable when dealing with
large datasets or continuous data streams. For this reason, large-scale AI projects
often require a more systematic and statistical approach to evaluation.

In such cases, annotated data serves as the reference point for the evaluation. Dur-
ing the annotation phase of an AI project, the dataset is typically divided into two
parts: a training set and a test set. The training set is used to teach the algorithm,
enabling it to recognize patterns, identify categories, and learn relationships between
variables. The test set, on the other hand, remains untouched during training and
serves as an independent benchmark to evaluate the model's performance once it has
been trained. This test set allows for the comparison of the algorithm’s outputs with the actual labels provided by human annotators, making it possible to measure the model’s accuracy. The test set serves as a so-called “gold standard”—a benchmark that represents the “ground truth” the algorithm must strive to reach (Jaton, 2017; Miceli et al., 2022). As such, human annotation is not only fundamental to the training phase but also remains central to the evaluation of the AI system, as it provides the reference point against which the AI’s performance is ultimately judged.

The gold standard allows for the calculation of various performance metrics, such as F1-score, precision score, recall score, and accuracy score. These metrics each capture different dimensions of the model’s performance, providing a nuanced understanding of how the algorithm is functioning. The F1-score, for example, offers an overall measure that takes into account both precision and recall, making it a useful indicator when both false positives (incorrectly identified elements) and false negatives (missed elements) are equally important. Precision focuses on the proportion of correctly identified elements out of all elements labelled by the algorithm, providing insight into how many of the model’s positive predictions are true. Recall, by contrast, measures the proportion of true elements that the model successfully identifies out of all relevant elements, making it critical in situations where capturing all relevant cases is more important than minimizing false positives.

The selection of performance metrics to prioritize depends heavily on the specific requirements and constraints of the AI project. For instance, when the risk associated with an algorithmic model error is considered high, alternative scores to the overall F1 performance score may be preferred. “This has been the case for the automatic anonymization of judicial decisions, in which missing a piece of sensitive information could have serious legal consequences, so capturing as much as possible is prioritized (Girard-Chanudet, 2023a, b). In this case, the focus has switched to maximizing recall, ensuring that all sensitive information is detected and anonymized, even if this means accepting a higher rate of false positives.

Overall, the process of evaluating an AI model using a gold standard and various performance metrics highlights the interplay between technical measurement and human judgement, and reaffirms the pivotal role played by annotation in the functioning of an AI model. From this point of view, the choice of an indicator remains a situated element, which only makes sense with regard to the specific structure of each production space. “Gold standard” thus appears as a constructed concept to be questioned, based on its underlying set of choices concerning perceived risks, and the data scientist’s representation of the model’s action.

## 4.2 From Models to Final Products: when AI Meets the Real World

In many cases, a simple human evaluation of model results based on a “gold standard” is not sufficient to assess whether a model functions as intended. In fact, several evaluation metrics are mobilized, depending on the objectives of the model designer. In addition to technical adequacy (can the model predict all ground-truth classes?), model designers evaluate models according to a set of more utilitarian criteria. For example, a scientist whose goal is to publish the results in a scientific journal will pay

<figure><img src="image_12.png" /></figure>
particular attention to the model’s ability to “leapfrog” existing models in predicting classes from an open-access “benchmark” dataset.

Similarly, when the model is integrated into a product, it is actually the produc-
tion chain of the “smart” service or product — in other words, the company’s ability
to produce reliably — that is being judged. Thus, if the technical shortcomings of a
model can be compensated for by adding humans to the loop, and if the growth pros-
pects are high and sustainable, then the product is a success, even if the accuracy of
the model is poor. Taken to its extreme, this approach leads some startups to produce
“false automation”-that is, to sell a service that is marketed as integrating an autono-
mous model but where annotators are the ones who provide real-time production.

> “There’s some literature on the topic, but pure computer science—let’s stick with the cliché of researchers—is mostly focused on fighting over improving neural networks to gain a few percentage points, rather than improving the data source itself. They’ll claim their neural network is better based on decimals gained from a dataset that’s not very good. But for us, from a business or engineering perspective, we don’t need to prove anything to anyone. If re-annotating a dataset improves performance, that’s what we’ll do instead.”

— Engineers, French AI company

Similarly, if a model has few or no directly marketable applications, but is part of a
“vertical” development perspective — in other words, if when added to other soft-
ware building blocks, it can represent an easily transposable service — it can be
considered a success.

Within a research context, the success of an algorithmic system is indexed on its
novelty (the gap in research filled) and its performance. The novelty of a model is
usually determined by comparing its performance to that of the existing state-of-the-
art models. These performances are themselves determined according to the accuracy
percentage reached on a benchmark dataset (as described above). The novelty of an
AI model enables professional recognition as a researcher.

This diversity of evaluation according to use cases and type of producer leads to a number of different scenarios. In particular, a model may be used in production even though it is considered technically inadequate. Similarly, a model may be considered technically efficient but have little or no “business value,” as is often the case with models developed in research and development departments. Thus, not all actors activate all levels of evaluation. Scientists, non-profit organizations, or startups in the incubation phase may be content to present proofs of concept that have no immediate business value. This is not the case for companies specializing in AI, especially “AI agencies”. For the latter, three main levels of evaluation are generally mobilized. In addition to the technical evaluation already mentioned, the model also goes through a business evaluation phase - is it mobilized and perceived as useful by employees, users or customers? - and strategic - is its use “profitable” or does it fit into a strategic development perspective for the company?

These levels of analysis quickly become blurred when it comes to evaluation with
model producers. As a result, metrics that describe model usage quickly become
more important than “accuracy” metrics. Evaluation of a final system or proof of concept into its envisioned environment tends to be conducted in quantified terms. At this point, the focus is often put on the model’s performance and its potential to be fine-tuned to a live scenario. Moreover, a system’s expected performance is necessarily based on a representation of its anticipated environment. Even if the latter appears highly realistic, there always remains a margin of error that is inevitable when quantifying organic or analog environments. Therefore, a gap will always remain between the sponsored narrative of a system’s goal and its actual possible use.

All of the mechanisms of evaluation and control discussed here remain accessible for the most parts for the producers of these systems. The lack of transparency that characterizes their development renders any public review quasi impossible by design. The most common critics thus take shape when failures occur and are made public – either AI systems perform differently than expected, either AI systems are put into use that transgress the average acceptability threshold. In this sense the public is rarely able to voice their opinion on AI systems.

# 5 Concluding Remarks

Based on the cross-analysis presented in this article, we have provided evidence of the distributed and iterative presence of evaluation throughout AI development, as summarized in the table below. While evaluation of AI system is often discussed as a monolithic task for which only a handful of qualified actors are responsible, we show how it is on the contrary a fragmented and multi-stakeholder process, as represented in Table 2. These plural evaluation practices play an essential role in shaping the AI system, which makes them essential to understand.

As summarized in Table 2, evaluation spans the entire chain that carries AI from its production to its deployment. The construction of the dataset required to train any algorithmic system entails the development and implementation of multiple evaluative strategies. Machine learning requesters must first define the objectives of their AI system, and accordingly determine the parameters of the training dataset. On this basis, the data annotators will organize and complete this work following several strategies of quality control, such as peer-review, performance reviews and economic incentives. These evaluative schemes are devised to optimize client satisfaction, which is in turn measured through different scales and methodologies. Once the final annotated dataset is delivered and approved, it will be used to train algorithmic models that will most likely serve as parts of more complex systems. These will generate new forms of evaluation and control depending on their claimed aims and envisioned uses, and might for instance be evaluated by users.

Evaluation is targeted at different objects and embodied in a diversity of activities, depending on the objectives of the actors using them and the point in the chain at which it takes place. In this context, multiple quality indicators coexist, providing numerous avenues for research to explore the interplay of activities and objectives within the algorithmic production process. Indicators indeed serve as a temporary focus for productive action — which by its very nature is negotiated among the actors involved — and their mobilization is accompanied by discourses that reveal
<table><caption>**Table 2** Presentation of the AI development roles and their associated evaluation practices</caption><thead><tr><th>Actor</th><th>Role</th><th>Mission statement</th><th>What is evaluated</th><th>Evaluation practices</th><th>Field-work</th></tr></thead><tbody><tr><td>Data annotators</td><td>Under-take the annotation</td><td>Deliver an annotation AI developers' expectations</td><td>Data ; Project feasibility ; Clients' requirements (payment, time allotted, etc.)</td><td>Routine evaluation of whether data / annotation guide meets the expectations of AI designers ; reviewing the annotation guide, asking questions to clients, Annotators cross-check in case of ambiguity of the Assess tasks based on previous experience</td><td>Fields 1; 2; 3</td></tr><tr><td>Quality analysts</td><td>Review annotation work</td><td>Deliver annotation that matches AI developers' expectations</td><td>Annotators' performance</td><td>Manual verification (sample or whole work) with quantitative and qualitative metrics</td><td>Fields 1; 2 and 3</td></tr><tr><td>Data work management executives</td><td>Translate the clients' needs into working project with annotation</td><td>Ensure the delivery of training sets that matches their expectations to clients</td><td>Project objectives, training sets, Economic feasibility of the project</td><td>Supervision of annotation teams with metrics / Client satisfaction</td><td>Field 1; 2 and 3</td></tr><tr><td>ML professionals (client)</td><td>Procure datasets for use</td><td>Procure highest quality dataset for lowest cost</td><td>Datasets. AI system, Model's business relevance</td><td>Dataset assessment ; algorithmic training quality</td><td>Fields 1; 2 and 3</td></tr></tbody></table>

the actor's; representations of the models — and the associated risks — that they are willing to consider. The distinction made in the introduction between a holistic and an “indicator-based” approach to evaluation is no longer so clear-cut. Actors mobilize different registers — and related artifacts — throughout the production chain, depending on the situation. As Burden (2024) points out, “indicator-based” approaches do not provide a global assessment of the model in use. We add that they offer an evaluation - situated in time - of specific aspects of it, produced for utilitarian purposes by actors who deploy strategies to secure their position within the various negotiation processes surrounding the model.

Empirical research from our three fields of study indeed allow us to show that the different actors involved in production assign varying values to the various elements mobilized in the process. Having a “good” annotation guide, for instance, means different things to annotators or data scientists. Moreover, these representations are not stable over time, and evolve throughout the model development process. This differentiated perception of the guide’s quality necessarily has an effect on the classes it describes, and by extension on the nature of the model. Here, we can take up the notion of “styles of valuation” defined by Francis Lee and Claes-Fredrik Helgesson (2020). Studying scientific production practices in a biological sciences laboratory where controversies surround the use of algorithmic methods for data analysis, the authors highlight the fact that “different configurations of algorithms, robots, and humans were valued in multiple ways in the same laboratory”, and that the circulation of different valuations, and their associated discourses, informs us about how actors represent their practices and legitimize them. In our article, we extend the
notion by claiming that these different styles of evaluation circulating within the pro-
duction apparatus and concerning the various artifacts and processes mobilized (data
sets, annotation guides, coordination meetings, etc.) are politically situated within
the strategies deployed by actors to maintain their position within the production
apparatus.

What actors understand by evaluation thus evolves according to their goals. While
we can agree with Milagros Miceli and Tianling Yang (2020) that the hierarchization
of production relationships between model designers and annotators actually gives
the former more power to define the framework of the production relationship and
impose the taxonomy they deem necessary to inscribe in the model, annotators retain
an active role in inscribing their practical knowledge in datasets. The structure of
outsourced data processing must therefore be taken into account in « capability-based
» assessments. Similarly, our analysis is coherent with the “regimes of counting”
studied by Chandhiramowuli and colleagues (2023), that allow AI clients to exert
control and authority over the annotation work undertaken by BPO companies. While
the counting practices they observe can be read as forms of evaluation, these authors’
focus is on the structuring effect of counting on the organisation of BPO-based data
work itself. Building on our empirical research, we have looked at a broader scope of
AI production, observing not only data annotation but also its integration and use in
AI systems development. By thus doing, our findings point to a more dialectical and
distributed understanding of evaluation of AI.

The model, taken at a given point in time, provides us with a crystallization of the
configuration of evaluation practices in place. Tracking the evaluations formulated
by actors and mapping the different styles of evaluation that are mobilized, as well
as the discourses and practices associated with them, thus allows us to take better
account of the issues surrounding the models and to better predict the direction in
which the models are heading, the reasons for this orientation and the different strate-
gies that actors are using to define this orientation, rather than through a prospective
study of the potential “risks” posed by the models themselves.

Claiming to assess so-called AI systems is not only a limited venture, it often
proves to be short-sighted in that it is usually based on a limited conception of these
systems. In fact, a cross-examination of the ways in which AI systems are developed
shows how their different parts are repetitively assessed throughout their production
by different evaluation methods. Therefore, meaningful evaluation of AI systems can
only occur if and when all of these practices are examined.

In this perspective, evaluating the model also implies evaluating the infrastructure
that supports it: the guidelines that shape annotation work, the training practices of
data workers, and the organizational settings in which such practices are embedded.
As previously discussed, recent comparisons between BPO and crowdsourcing mod-
els (Rädsch et al., 2024) further support the idea that quality is not only a property
of output, but a result of socially and materially situated production arrangements.

Tracking actors throughout the chain in their evaluation practices thus makes it
possible to link the two aspects of evaluation: by bringing together all evaluation
styles, whether indicator-based or not, we can map the issues that actors place within
the model and appreciate what is left out, deliberately or otherwise, when we take an
interest in the model. If we take the case of Open AI mentioned in the introduction,

<figure><img src="image_13.png" /></figure>
the risks in terms of misinformation left out at the time of the model’s launch, rather than being a technical inevitability linked to the limitations of large language models, result from a decision on resource allocation and operational design at the time of the model’s conception - which, moreover, is constantly evolving. Mapping the representations that actors have of the model and the values they assign to it - which necessarily involves mapping the model’s production chain in order to identify the actors involved and the issues surrounding production - helps to contextualise the choices made when considering the risks of the model.

The intersection between valuation studies and digital labour studies that we have proposed in this article provides insights for analyzing how AI systems take shape through successive, continuous processes of assessment, which lead to gradual adjustments and back-and-forth movements along the production chain. Focusing on evaluation—what is being evaluated, the indicators used within this framework, and the diverse actors involved in these processes—allows for a fine-grained identification of the values embedded in algorithms, the objectives they serve, and, in essence, their political and social implications. We argue that evaluation is, first and foremost, a form of labour, involving a wide range of actors beyond economic decision-makers or technical experts. These actors, whose choices and goals shape AI tools, are crucial in the making of AI.

The iterative, process-based nature of algorithm construction means that regulatory intervention is possible at many points in the chain. In this sense, acting on the “existential risks” (Critch & Krueger, 2020) posed by AI can therefore start with greater transparency on how the latter are produced, by making annotation guides available, and making outsourced production chains visible. But beyond transparency, intervention can also involve the opening of alternative evaluation regimes that could come into play at each step of the process. This includes integrating equity and diversity indicators at the goal-setting stage, shaping classification systems by drawing on annotators’ expertise, and conducting impact assessments during model evaluation to better anticipate and mitigate potential harms.

In conclusion, effective regulation should foster transparency not only in the supply chain—by ensuring visibility into subcontractors’ identities and locations—but also in production processes. This includes the accessibility of guidelines, their design history, and managerial procedures. By enhancing both dimensions of transparency, regulatory frameworks can promote accountability, improve compliance, and ultimately contribute to more ethical and sustainable production practices. Therefore, we believe that the approach should not be purely technical, as this may overlook the production stages of the models, nor should it be overly flexible, at least regarding data manufacturing methods.

**Acknowledgements** The authors would like to thank all the data workers who kindly agreed to participate in the various fieldworks used in this article.

**Funding** Open access funding provided by Université Paris-Saclay. This article presents the results of the project HUSH ‘The Human Supply Chain Behind Smart Technologies’ funded by the French National Research Agency (Project ANR-19-CE10-0012). This research also benefited from an additional funding provided by a French AI start-up in order to pay interviewees.
**Data Availability** Not applicable.

**Declarations**

**Conflicts of Interest** The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: An anonymous start-up provided an additional source of funding to allow access to their workforce.

**Open Access** This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

**References**

Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016, May 23). *Machine bias—There's software used across the country to predict future criminals. And it's biased against blacks*. ProPublica, Online Edition.

Bechmann, A., & Bowker, G. C. (2019). Unsupervised by any other name: Hidden layers of knowledge production in artificial intelligence on social media. *Big Data & Society*, *6*(1), 2053951718819569. https://doi.org/10.1177/2053951718819569

Bowker, G.C., Star, S.L., (1999). Sorting things out: classification and its consequences. The MIT Press, Cambridge. https://doi.org/10.7551/mitpress/6352.001.0001

Buolamwini, J., & Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. *Proceedings of the 1st Conference on Fairness, Accountability and Transparency*, 77–91. https://proceedings.mlr.press/v81/buolamwini18a.html

Burden, J. (2024). Evaluating AI evaluation: Perils and prospects (No. arXiv:2407.09221). arXiv. https://doi.org/10.48550/arXiv.2407.09221

Burrell, J. (2016). How the machine 'thinks': Understanding opacity in machine learning algorithms. *Big Data & Society*, *3*(1), 2053951715622512. https://doi.org/10.1177/2053951715622512

Chandhiramowuli, S., Taylor, A., Heitlinger, S., & Wang, D. (2023). Making data work count. arXiv. https://doi.org/10.48550/arXiv.2311.18046

Chmielewski, M., & Kucker, S. C. (2020). An MTurk crisis? Shifts in data quality and the impact on study results. *Social Psychological and Personality Science*, *11*(4), 464–473. https://doi.org/10.1177/1948550619875149

Christin, A. (2020). The ethnographer and the algorithm: Beyond the black box. *Theory and Society*, *49*(5), 897–918. https://doi.org/10.1007/s11186-020-09411-3

Critch, A., & Krueger, D. (2020). AI research considerations for human existential safety (ARCHES) (No. arXiv:2006.04948). arXiv. https://doi.org/10.48550/arXiv.2006.04948

Daston, L., & Galison, P. (2010). *Objectivity*. Zone Books.

Derczyński, L. (2016). Complementarity, F-score, and NLP Evaluation. In N. Calzolari, K. Choukri, T. Declerck, S. Goggi, M. Grobelnik, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, & S. Piperidis (Eds.), *Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)* (pp. 261–266). European Language Resources Association (ELRA). https://aclanthology.org/L16-1040/.

Floridi, L., & Strait, A. (2020). Ethical foresight analysis: What it is and why it is needed? *Minds and Machines*, *30*(1), 77–97. https://doi.org/10.1007/s11023-020-09521-y

Frey, C. B., & Osborne, M. A. (2017). The future of employment: How susceptible are jobs to computerisation? *Technological Forecasting and Social Change*, *114*, 254–280. https://doi.org/10.1016/j.techfore.2016.08.019
Gabriel, I. (2020). Artificial intelligence, values and alignment. *Minds and Machines*, 30(3), 411–437.
https://doi.org/10.1007/s11023-020-09539-2

Girard-Chanudet, C. (2023a). *La justice algorithmique en chantier. Sociologie du travail et des infra-structures de l'Intelligence Artificielle* [Phdthesis, EHESS (École des Hautes Études en Sciences Sociales)]. https://hal.science/tel-04398889

Girard-Chanudet, C. (2023b). « Mais l’algo, Là Il Va mimer Nos erreurs ! » Contraintes et effets de l’annotation des Données d’entrainement d’une IA. *Réseaux*, 4/240, 111–144.

Gornet, M., & Herman, H. (2024). A peek into European standards making for AI: Between geopolitical and economic interests. https://hal.science/hal-04784035

Gray, M. L., & Suri, S. (2019). *Ghost work: How to stop Silicon Valley from building a new global under-class*. Houghton Mifflin Harcourt.

Helgesson, K., & Muneisa, F. (2013). For what it's worth: An introduction to valuation studies. *Valuation Studies*, 1(1), 1–10.

Hendrycks, D., Mazeika, M., & Woodside, T. (2023). An overview of catastrophic AI risks (No. arXiv:2306.12001). *arXiv*. https://doi.org/10.48550/arXiv:2306.12001

Irani, L. (2015). Difference and dependence among digital workers: The case of Amazon mechanical Turk. *South Atlantic Quarterly*, 114(1), 225–234. https://doi.org/10.1215/00382876-2831665

Jaton, F. (2017). We get the algorithms of our ground truths: Designing referential databases in digital image processing. *Social Studies of Science*, 47(6), 811–840. https://doi.org/10.1177/0306312717730428

Jaton, F. (2021). The constitution of algorithms ground-truthing, programming, formulating. https://doi.org/10.7551/mitpress/12517.001.0001

Jobin, A., Ienca, M., & Vayena, E. (2019). The global landscape of AI ethics guidelines. *Nature Machine Intelligence*, 1(9), 389–399. https://doi.org/10.1038/s42256-019-0088-2

Knorr-Cetina, K. (1999). *Epistemic cultures: How the sciences make knowledge*. Harvard University Press.

Krijger, J. (2024). What about justice and power imbalances? A relational approach to ethical risk assessments for AI. *Digital Society*, 3(3), 56. https://doi.org/10.1007/s44206-024-00139-6

Laux, J., Stephany, F., & Liefgreen, A. (2023). The economics of human oversight: how norms and incentives affect costs and performance Of AI workers (No. arXiv:2312.14565). *arXiv*. https://doi.org/10.48550/arXiv.2312.14565

Lebovitz, S., Levina, N., & Lifshitz-Assaf, H. (2021). Is AI ground truth really ‘true’? The dangers of training and evaluating AI tools based on experts’ Know-What (SSRN scholarly paper ID 3839601). *Social Science Research Network*. https://papers.ssrn.com/abstract=3839601

Lee, F., & Helgesson, C. F. (2020). Styles of valuation: Algorithms and agency in high-throughput bioscience. *Science Technology & Human Values*, 45(4), 659–685.

Le Ludec, C., Cornet, M., & Casilli, A. A. (2023). The problem with annotation. Human labour and outsourcing between France and Madagascar. *Big Data & Society*, 10(2), 20539517231188723. https://doi.org/10.1177/20539517231188723

Ludec, C. L. (2024). *Des humains derrière l'intelligence artificielle. La sous-traitance du travail de la donnée entre la France et Madagascar* [Phdthesis, Institut Polytechnique de Paris]. https://hal.scie-nce/tel-04605206

Miceli, M., Posada, J., & Yang, T. (2022). Studying up machine learning data: Why talk about bias when we mean power? *Proceedings of the ACM on Human-Computer Interaction*, 6(GROUP), 341–3414.
https://doi.org/10.1145/3492853

Miceli, M., Schuessler, M., & Yang, T. (2020). Between subjectivity and imposition: Power dynamics in data annotation for computer vision. *Proceedings of the ACM on Human-Computer Interaction*, 4(CSCW2), 1–25. https://doi.org/10.1145/3415186

Muldoon, J., Cant, C., & Wu, B., & Graham, M. (2024). et. « A Typology of Artificial Intelligence Data Work. *Big Data & Society* 11(1), (1 mars 2024): 20539517241232632. https://doi.org/10.1177/20539517241232632

Muniesa, F. (2003). *Des marchés comme algorithmes: Sociologie de la cotation électronique à la bourse de Paris* [These de doctorat, Paris, ENMP]. https://theses.fr/2003ENMP1163

Pasquale, F. (2015). *The black box society: The secret algorithms that control money and information*. Harvard University Press.

Polanyi, M. (1958). Personal knowledge: Towards a post-critical philosophy. In M. J. Nye (Ed.), *University of Chicago Press*.

Rädsch, T., Reinke, A., Weru, V., Tizabi, M. D., Heller, N., Isensee, F., Kopp-Schneider, A., & Maier-Hein, L. (2024). Quality assured: Rethinking annotation strategies in imaging AI (arXiv:2407.17596). *arXiv*. https://doi.org/10.48550/arXiv.2407.17596
Schmidt, F. A. (2019). *Crowdproduktion von Trainingsdaten: Zur Rolle von Online-Arbeit beim Trainieren autonomer Fahrzeuge* (Research Report 417). Study der Hans-Böckler-Stiftung. https://www.econs tor.eu/handle/10419/201850

Seaver, N. (2017). *Algorithms as culture: Some tactics for the ethnography of algorithmic systems*. *Big Data & Society*, 4(2), 2053951717738104. https://doi.org/10.1177/2053951717738104

Tubaro, P., & Casilli, A. A. (2019). Micro-work, artificial intelligence and the automotive industry. *Journal of Industrial and Business Economics*, 46(3), 333–345. https://doi.org/10.1007/s40812-019-00121-1

Tubaro, P., Casilli, A. A., & Coville, M. (2020). The trainer, the verifier, the imitator: Three ways in which human platform workers support artificial intelligence. *Big Data & Society*. https://doi.org/10.1177 /2053951720919776

Vatin, F. (2013). Valuation as evaluating and valorizing. *Valuation Studies*, 1(1). https://doi.org/10.3384/ vs.2001-5992.131131

Veselovsky, V., Ribeiro, M. H., & West, R. (2023). Artificial artificial artificial intelligence: crowd workers widely use large language models for text production tasks (No. arXiv:2306.07899). *arXiv*. https://d oi.org/10.48550/arXiv.2306.07899

Vold, K., & Harris, D. R. (2023). How Does Artificial Intelligence Pose an Existential Risk? In C. Véliz (Ed.), *Oxford handbook of digital ethics* (p. 0). Oxford University Press. https://doi.org/10.1093/oxf ordhb/9780198857815.013.36

Webb, Margaret A. & June P. Tangney (2022). Too good to be true: Bots and Bad Data from Mechanical Turk. *Perspectives on Psychological Science*.19, 6. https://doi.org/10.1177/17456916221120027

**Publisher’s Note** Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.