<figure><img src="image_1.png" /></figure>

HAL
open science

# Human-Centered Risk Governance for Adaptive AI

Alexandru Mateescu

► To cite this version:

Alexandru Mateescu. Human-Centered Risk Governance for Adaptive AI. European Workshop on Trustworthy AI (TRUST-AI 2025), European Conference on Artificial Intelligence (ECAI 2025), Oct 2025, Bologna Italy, Italy. hal-05427990

HAL Id: hal-05427990

https://hal.parisnanterre.fr/hal-05427990v1

Submitted on 21 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Human-Centered Risk Governance for Adaptive AI: Why Educational Requirements Belong in Trustworthy AI Frameworks

Alexandru Mateescu¹

¹Université Paris 1 Panthéon-Sorbonne, École Doctorale 280, Department of Philosophy, Paris, France

## Abstract

Adaptive AI systems — those that evolve through user interaction — create distinctive epistemic risks: users may lose track of how the system changes, why outputs are produced, or how their actions shape future recommendations. These risks go beyond standard concerns such as fairness or robustness and are not addressed by existing safeguards. We argue that trust in such systems should be understood as a relational process involving both users and designers. From this perspective, we introduce educational requirements: lifecycle design commitments embedded throughout the AI system to promote user intelligibility, reflection, and control. This approach aligns with the Human-Centered AI perspective, which emphasizes combining high levels of automation with equally high levels of human control to ensure reliable, safe, and trustworthy systems [1].

Beyond delivering information, these requirements must also ensure that educational content is received and acknowledged. Emerging content management technologies — including blockchain-based approaches — could soon enable the tracking of distribution and uptake of key educational snippets, providing a verifiable record of user engagement and accountability.

Grounded in regulatory frameworks such as the AI Act and the Digital Services Act, these requirements offer a proportionate and forward-looking way to mitigate epistemic risks and support human-centered AI governance.

## 1. Introduction

Adaptive AI systems — those that evolve through user interaction — are now a central concern for regulators. Frameworks such as the NIST AI Risk Management Framework [2], ENISA guidelines [3], and the European AI Act [4] emphasize the need for oversight of systems that continue to learn after deployment.

Adaptivity introduces distinctive risks. When AI systems evolve by responding to user behavior — as in many recommender contexts — they also shape that behavior in return. This feedback loop creates epistemic risks: users may lose track of how the system changes, why outputs are produced, or how their actions influence future recommendations.

These dynamics challenge conventional models of trustworthiness. Existing safeguards such as fairness, explainability, and robustness are necessary but insufficient. Trust must instead be understood as a relational process co-constructed by users and designers over time.

We propose educational requirements: design commitments embedded throughout the AI lifecycle to promote user intelligibility, reflection, and control. Grounded in emerging regulations such as the AI Act and the Digital Services Act, these requirements offer a proportionate way to mitigate epistemic risks and support more accountable, human-centered AI governance.

## 2. Adaptivity and Epistemic Risk

Adaptive AI systems do not follow fixed rules: they continuously adjust their outputs based on user interactions and behavioral patterns. While this improves personalization, it also creates distinctive epistemic risks. Users may lose track of how the system evolves, why certain outputs are produced, or how their actions shape future recommendations.

TRUST-AI Workshop, ECAI 2025, Bologna, Italy — October 25, 2025

<figure><img src="image_3.png" /></figure>

alexandru.mateescu@etu.univ-paris1.fr (A. Mateescu)

<figure><img src="image_4.png" /></figure>

© 2025 This work is licensed under a "CC BY 4.0" license.
These risks differ from conventional concerns such as fairness or robustness. They undermine the user's ability to interpret and influence a system whose logic shifts over time. The problem is most acute when users are treated merely as variables in optimization, rather than as epistemic agents capable of making sense of the process.

This opacity can foster harms such as biased beliefs, dependency, or disengagement. It also challenges regulatory goals like transparency, which presume that users can understand and contest significant automated decisions [4].

Addressing these risks requires mechanisms that go beyond technical transparency and documentation. Prior work in VIS4ML highlights that many interactive ML systems remain limited to case studies and lack verifiable evidence of real-world impact, underscoring the need for governance approaches that integrate verification and accountability [5]

### 3. Framing Trust as a Relational Epistemic Process

Standard approaches to trustworthy AI focus on system attributes such as fairness, accuracy, robustness, or explainability. These are essential but reflect an internalist view, where trustworthiness is judged solely by the system's properties. In adaptive contexts, this view is insufficient.

Trust must instead be seen as relational: an evolving process between users and systems that themselves evolve through interaction. Adaptive AI not only learns from users but also shapes their behavior. Trust depends on the user's ongoing capacity to interpret, evaluate, and contest this dynamic.

Promoting trustworthiness therefore requires supporting the user's epistemic role, not just improving algorithms. This involves making adaptation mechanisms visible and giving users tools to track and influence outcomes.

This perspective aligns with regulatory goals in the AI Act [4] and the Digital Services Act [6], which emphasize transparency and oversight. Implementing these principles requires more than documentation: it calls for designs that embed users in the learning loop as interpreters and co-regulators, not merely data points.

The next section introduces educational requirements as a practical way to operationalize this relational approach and mitigate epistemic risks.

### 4. Educational Requirements as Governance Mechanisms

Educational requirements are system-level design commitments that make adaptivity intelligible to users and actionable for designers. Unlike external documentation or user training, they are embedded directly into the interface, feedback loops, and lifecycle processes of an adaptive AI system. Their purpose is to ensure that as the system evolves, users remain active epistemic agents rather than passive data sources.

We distinguish between two roles:

1. **End users** – whose interactions generate the data that drive adaptation.

2. **Designers and developers** – who define the mechanisms of adaptation and must anticipate the epistemic risks they create.

Failures arise when these roles are misaligned — for instance, when a recommender system is optimized to maximize engagement while users seek informed decisions [7]. Educational requirements help bridge this gap by embedding mutual intelligibility into the design process.

Examples from recent work illustrate how educational interventions can be embedded directly into recommender systems to foster user understanding and agency [8, 9].

Explanations alone are insufficient: research shows they reduce overreliance only for relatively easy decisions where users can already verify the correct answer, but fail to support decision-making in harder cases [10]. Concretely, these requirements can take several forms:
* **Traceability indicators** – showing how current outputs relate to past interactions and how future recommendations may shift as a result.

* **Dynamic transparency prompts** – surfacing which signals the system is prioritizing, reducing the “black-box” effect of invisible adaptation.

* **Feedback channels** – allowing users to contest or redirect system learning when outputs misalign with their goals.

* **Verification of educational uptake** – mechanisms that assess whether key educational snippets have been delivered and processed by the user. Emerging content management capabilities, including blockchain-based approaches, make it possible to track the distribution and acknowledgment of such snippets while preserving accountability and user rights.

Implementing these mechanisms raises practical challenges. Excessive detail risks overwhelming users, while oversimplification fosters false confidence. Commercial incentives may also conflict with the goal of user education, as platforms often benefit from opacity or frictionless engagement. Addressing these tensions requires deliberate trade-offs, balancing intelligibility, usability, and business goals.

Because they can be integrated at design, development, and testing stages, educational requirements are compatible with lifecycle governance. They operationalize human-centered oversight, ensuring that epistemic risks are treated not as side effects but as core governance targets, alongside explainability and fairness.

# 5. Regulatory Alignment and Proportionality

Educational requirements are not only ethically motivated — they also align with emerging legal frameworks. The Digital Services Act (DSA) introduces the concept of systemic risks, including threats to civic discourse and individual agency, and emphasizes proportional mitigation obligations [6]. Similarly, the AI Act calls for governance measures tailored to the context and risk level of specific systems [4]. Adaptive AI systems, especially recommender systems, often fall into gray zones: they shape user behavior in subtle ways that may not trigger strict legal thresholds but still cause epistemic harm. Educational requirements provide a balanced alternative to both heavy-handed intervention and passive transparency. They allow designers to address these risks proactively without limiting system capabilities or imposing intrusive oversight.

Because they can be integrated at the stages of design, development, and testing, educational requirements fit naturally into compliance checklists, certification schemes, and audits as the AI Act moves toward enforcement. By making the user's epistemic position a measurable aspect of trustworthiness, they extend the principle of proportionality into the cognitive domain, ensuring that adaptive systems support not only legal compliance but also the user's right to understand and shape algorithmic processes over time. The principle of proportionality, central to both the AI Act and the DSA, ensures that governance measures are calibrated to the level of risk: more stringent obligations for systems with greater potential harm, and lighter measures for those with minimal impact.

# 6. Conclusion

Adaptive AI systems create new forms of epistemic risk that conventional safeguards such as fairness, explainability, or robustness cannot fully address. These risks emerge from feedback loops where users both influence and are influenced by system behavior, often without clear visibility into this process.

We have argued that trust in adaptive AI should be understood as a relational process involving both users and designers. From this perspective, we proposed educational requirements: lifecycle design commitments that embed intelligibility and user agency directly into adaptive systems.

Crucially, these requirements must go beyond simply presenting information. Systems should be able to assess whether educational snippets have been delivered and engaged with, ensuring that users are not only exposed to content but also empowered to act on it. Emerging content management capabilities,
potentially supported by blockchain technologies, could soon allow for verifiable records of distribution and acknowledgment. This would strengthen both accountability and regulatory compliance while preserving user rights.

By aligning with the AI Act and Digital Services Act, educational requirements become a proportionate governance mechanism, bridging the gap between formal legal compliance and meaningful user oversight. We invite researchers, designers, and regulators to explore and refine these requirements, and to collaborate on developing the technological foundations — including content management and blockchain-based tracking — needed to verify the delivery and acknowledgment of educational snippets. Working together, we can ensure that adaptive AI systems remain transparent, intelligible, and accountable as they evolve.

## References

[1] B. Shneiderman, Human-centered artificial intelligence: Reliable, safe & trustworthy, arXiv preprint arXiv:2002.04087 (2020). URL: https://arxiv.org/abs/2002.04087.

[2] NIST, Ai risk management framework (ai rmf 1.0), https://www.nist.gov/itl/ai-risk-management-framework, 2023. Accessed: 2025-09-20.

[3] ENISA, Artificial intelligence cybersecurity challenges, https://www.enisa.europa.eu/publications/artificial-intelligence-cybersecurity-challenges, 2020. Accessed: 2025-09-20.

[4] E. Commission, Artificial intelligence act – final political agreement, https://artificial-intelligence-act.eu/, 2024. Accessed: 2025-09-20.

[5] H. Subramonyam, J. Hullman, Are we closing the loop yet? gaps in the generalizability of vis4ml research, IEEE Transactions on Visualization and Computer Graphics 29 (2023) 1–15. doi:10.1109/TVCG.2023.3243440.

[6] E. Commission, Digital services act, https://digital-strategy.ec.europa.eu/en/policies/digital-services-act-package, 2022. Accessed: 2025-09-20.

[7] N. Seaver, Captivating algorithms: Recommender systems as traps, Journal of Material Culture 28 (2023) 3–21. doi:10.1177/13591835221081874.

[8] L. Alves, R. Cardoso, R. B. C. Prudêncio, Digital nudges for recommender systems: Managing exploration and exploitation trade-offs, Information Processing & Management 61 (2024) 103269. doi:10.1016/j.ipm.2023.103269.

[9] V. Lomonaco, D. Maltoni, et al., Game-based education to address filter bubbles and echo chambers, in: Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, ACM, 2022, pp. 905–917. doi:10.1145/3531146.3534634.

[10] Z. T. Zhang, F. Buchner, Y. Liu, A. Butz, You can only verify when you know the answer: Feature-based explanations reduce overreliance on ai for easy decisions, but not for hard ones, in: Proceedings of Mensch und Computer 2024 (MuC '24), 2024, pp. 1–15. URL: https://doi.org/10.1145/3670653.3670660. doi:10.1145/3670653.3670660.