Artificial intelligence and the rights of the child : towards an integrated agenda for research and policy.
Date: 2022-06-07
Institutions: Joint Research Centre
Subjects: EU action, EU research policy, EU strategy, European audiovisual area, artificial intelligence, children's rights, data protection, distance learning, education, information security, policymaking, protection of privacy, report, research programme, robotics, scientific report, social integration

Artificial Intelligence and 
the Rights of the Child
Towards an Integrated Agenda for Research and Policy 

Charisi, V., Chaudron, S., Di Gioia, 
R., Vuorikari, R., Escobar-Planas, M., 
Sanchez, I., Gomez, E.

2022 

EUR 31048 EN

 
 
 
 
 
 
 
 
About JRC

This publication is a Science for Policy report by the Joint Research Centre (JRC), the European Commission’s 
science and knowledge service. It aims to provide evidence-based scientific support to the European policymaking 
process. The scientific output expressed does not imply a policy position of the European Commission. Neither 
the European Commission nor any person acting on behalf of the Commission is responsible for the use that 
might be made of this publication. For information on the methodology and quality underlying the data used 
in this publication for which the source is neither Eurostat nor other Commission services, users should contact 
the referenced source. The designations employed and the presentation of material on the maps do not imply 
the expression of any opinion whatsoever on the part of the European Union concerning the legal status of any 
country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries.

Contact information: 
Vasiliki Charisi, European Commission, Joint Research Centre, Seville - SPAIN 
vasiliki.charisi@ec.europa.eu  

EU Science Hub: https://joint-research-centre.ec.europa.eu/

JRC127564

EUR 31048 EN

PDF 

ISBN 978-92-76-51837-2  

ISSN 1831-9424  

doi:10.2760/012329

Luxembourg: Publications Office of the European Union, 2022

© European Union, 2022

The reuse policy of the European Commission is implemented by the Commission Decision 2011/833/EU of 
12 December 2011 on the reuse of Commission documents (OJ L 330, 14.12.2011, p. 39). Except otherwise 
noted,  the  reuse  of  this  document  is  authorised  under  the  Creative  Commons  Attribution  4.0  International 
(CC BY 4.0) licence (https://creativecommons.org/licenses/by/4.0/). This means that reuse is allowed provided 
appropriate  credit  is  given  and  any  changes  are  indicated.  For  any  use  or  reproduction  of  photos  or  other 
material that is not owned by the EU, permission must be sought directly from the copyright holders.

All content © European Union, 2022, except: p. IV ©metamorworks_AdobeStock_492532786; p. 4 ©Have a 
nice day_ Futuristic Robotic creative learning education school_AdobeStock_377608563; p. 6 ©boy_with_ro-
bot_(c)_chika_milan_488569153;  p.  10  ©denisismagilov_kid  boy  with  tablet_AdobeStock_491007420;  p. 
34 ©zinkevych_favorite activity_AdobeStock_201497369; p. 44 ©hakule_litter girl toucing the future_Ado-
beStock_497431668;  p.  58  ©Danai_Virtual  reality  eyewear_AdobeStock_502204437;  p.  62  ©metamor-
works_AdobeStock_484353731.

How to cite this report: Charisi, V., Chaudron, S., Di Gioia, R., Vuorikari, R., Escobar-Planas, M., Sanchez, I., Gomez, 
E.  (2022)  Artificial  Intelligence  and  the  Rights  of  the  Child:  Towards  an  Integrated  Agenda  for  Research  and 
Policy, EUR 31048 EN, Publications Office of the European Union, Luxembourg, 2022, ISBN 978-92-76-51837-2, 
doi:10.2760/012329, JRC127564.

 
 
Authors

This  report  was  authored  by  Vicky  Charisi,  Stéphane  Chaudron,  Rosanna  Di  Gioia,  Riina  Vuorikari, 
Marina Escobar-Planas, Ignacio Sanchez and Emilia Gómez. 

DG.JRC.B6 - Digital Economy 
DG.JRC.E3 - Cyber and Digital Citizen’s Security 
DG.JRC.B4 - Human Capital and Employment

Acknowledgements

This report is the result of a collective effort of an interdisciplinary and multi-stakeholder team of 
children and young people, experts in the field of AI and child’s rights, policymakers from various 
Directorates-General  of  the  European  Commission  and  other  policy-oriented  organisations  and 
researchers from the JRC.

Experts

Victoria Baines, University of Bournemouth, UK  
Tara Chklovski, Technovation, US 
Davinia Hernandez-Leo, Universitat Pompeu Fabra (UPF), Barcelona, ES 
Giovanna Mascheroni, Università Cattolica del Sacro Cuore di Milano, IT  
Soledad Pera, Boise State University, US  
Jochen Peter, Universiteit van Amsterdam (UvA), NL 
Valerie Verdoot, Universiteit Gent (UGent), BE 
Bieke Zaman, Katholieke Universiteit Leuven (KUL), BE 

Young BIK (Better Internet for Kids) Panelists

We warmly thank the Young BIK (Better Internet for Kids) Panellists – Darina, Edgars, Jorge, 
Mateij, Molly, Neda, Lorcan, Solena, Sunna, Violet, Yevgeny – who participated actively and with 
enthusiasm in the workshops* that were the basis of this work.

And their coordinators and assistant, Sabrina Vorbau and Eray Basar (INSAFE – European Schoolnet, led by 
Hans Martens) and the 8 members of the Spanish BIK Youth Ambassador (coordinated by Plataforma de 
Infancia) who provided additional considerations on the issue of AI and children’s rights.

European Commission and Executive Agencies

DG.CNECT.G.3 - Accessibility, Multilingualism and Safer Internet: Manuela Martra, Donatella  
Nembrini, Anna Sobkiewicz, Margarita Akritidou 
DG.HOME.D4 - Security in the digital age: Filipa Vilela 
DG.JUST.C.2 - Fundamental Rights Policy: Elizabeth Quinn 
DG.EAC.C4 - Digital Education: Maria Gkountouma 
DG.JRC.E.3 - Cyber and Digital Citizen’s Security: Massimiliano Gusmini, Gillian O’Neill, Guido Schillaci.  
FRA - European Union Agency for Fundamental Rights: Monica Gutierrez Arques, Gerard Rosskogler

* For a detailed description of the youth workshops, please refer to Chaudron, S. and Di Gioia, R., Artificial Intelligence 
and the Rights of the Child - Young people’s views and perspectives, EUR 31094 EN, Publications Office of the European 
Union, Luxembourg, 2022, ISBN 978-92-76-53101-2, doi:10.2760/272950, JRC129099. 
https://publications.jrc.ec.europa.eu/repository/handle/JRC129099

II

 
 
TABLE OF CONTENTS

ABSTRACT 

EXECUTIVE SUMMARY 

Structure of the report 

1. INTRODUCTION 

2. EXISTING POLICY INITIATIVES ON AI AND CHILDREN 

2.1 The UN Convention on the Rights of the Child 

2.2 The UN General Comment 25 

2.3 The UNICEF initiative on AI and children’s rights 

2.4 OECD related initiatives 

2.5 UNESCO related initiatives 

2.6 IEEE related initiatives 

2.7 Council of Europe related initiatives 

2.8 European Commission related initiatives 

2.8.1 The European strategy on the rights of the child 

2.8.2 Artificial intelligence policies at the European Commission 

2.8.3 The Digital Education Action Plan and audio-visual media services directive 

3. ANALYSIS OF THE IMPACT OF SELECTED AI APPLICATIONS ON  
    CHILDREN’S RIGHTS 

3.1 Recommender systems 

3.2 Conversational agents 

3.3 Robotic systems 

4. YOUTH’S PERSPECTIVES: WORKSHOPS WITH CHILDREN AND  
    YOUNG PEOPLE 

4.1 Participants’ preconceptions of AI 

4.2 Participants’ perceptions on the impact of AI on children’s rights 

4.3 What do young people tell us, what are their concerns? 

4.4 What do young people propose? 

4.5 Participants’ questions about future directions 

5. EXPERTS’ PERSPECTIVES: FROM WORKSHOPS WITH  
    SCIENTISTS AND POLICYMAKERS 

1

2

5

7

11

12

12

13

14

15

16

17

18

18

20

21

23

26

28

30

35

36

37

41

41

42

45

IIII

5.1 Requirements for AI policy supporting children’s rights 

5.1.1 AI Minimisation, valuable purposes and sustainability 

5.1.2 Transparency, explainability and accountability 

5.1.3 Inclusion and non-discrimination 

5.1.4 Privacy, data protection and safety 

5.2 Methods to develop AI policy supporting children’s rights 

5.2.1 Anticipation, evaluation and monitoring 

5.2.2 Multistakeholder collaboration 

5.2.3 Children’s participation 

5.2.4 Balancing conflicting rights 

5.3 Identified knowledge gaps requiring further research 

5.3.1 Children’s cognition, development, and play 

5.3.2 Empowering through education 

5.3.3 Developmentally appropriate systems and age verification 

5.3.4 Other research questions to be addressed 

6. RESEARCHERS, POLICYMAKERS AND CHILDREN:  
    A TRIANGULATION OF PERSPECTIVES 

7. LIMITATIONS AND CONSIDERATIONS 

8. CONCLUSIONS 

9. FOR DEEPER CONSIDERATION - REFLECTIONS OF INVITED EXPERTS 

9.1 Regulation and policies on artificial intelligence should consider the full range  
      of children’s rights, especially when these conflict 

9.2 AI education: Beyond mere transfer of knowledge 

9.3 Directions for the responsible design and use of AI by children and their  
      communities: Examples in the field of Education 

9.4 Children’s identities and partial algorithmic identifications: A gap to bridge  
      with human agency 

9.5 Evaluating and monitoring AI applications in the classroom: Open Challenges 

9.6 Children’s right to protection from economic exploitation in an AI-world 

REFERENCES 

LIST OF FIGURES 

LIST OF TABLES 

46

46

47

47

49

51

51

51

51

53

54

54

55

56

56

59

63

65

69

70

72

73

75

76

77

79

87

87

IIIIII

IVIV

ABSTRACT

Artificial Intelligence undoubtedly offers children many opportunities, but it also raises the risk of 
compromising their rights. While an increasing number of policy initiatives and corresponding research 
seek to better understand and provide solutions to mitigate the risks and augment the benefits of 
AI-based technologies for children, there is often a lack of interaction among stakeholders. 

This report provides elements for an integrated agenda for research and policy regarding children’s 
rights  and  Artificial  Intelligence.  It  aims  to  connect  scientific  evidence  with  policymaking,  to  gain 
insights  of  the  interplay  between  different  stakeholders,  and  to  go  beyond  the  identification  of 
ethical guidelines towards methods for practical future implementations. 

A review of relevant policy initiatives is presented. This is followed by an overview of research on 
three  selected  AI  applications  for  children,  namely  conversational  agents,  recommender  systems 
and robotic systems, under the lens of children’s rights. Then, the results of two workshops with 
children and young people, and three workshops with stakeholders from policy and research in the 
field of AI and child’s rights are presented. A thematic analysis of the discussions revealed different 
sets of topics proposed by the stakeholders, which have then been triangulated to identify priorities 
from the different perspectives.

Based  on  the  above-mentioned  input,  a  set  of  recommendations  for  an  integrated  agenda  and 
future  directions  in  research  and  policy  are  provided.  Lastly,  this  report  includes  reflections  from 
invited experts in the field, who participated and contributed to this study.

11

EXECUTIVE SUMMARY

Children’s fundamental rights when using 
AI-based technology

Requirements, methods and knowledge-
gaps

global 

technology 

Intelligence-based 

Artificial 
is 
increasingly  present  in  our  children’s  lives.
and 
Governments 
organisations 
initiatives  and  establishing 
are  developing 
frameworks  to  create  trustworthy  and  child-
centred AI technology. Significant steps in policy 
and research have already been taken to identify 
the core elements needed for trustworthy AI for 
children. 

However,  stakeholders  from  the  worlds  of 
institutions,  and 
academia,  policymaking 
industry  mostly  follow  their  own  agendas, 
with 
little  awareness  of  the  needs  and 
values  of  others.  These  communities  do  not 
always  have  the  opportunity  to  work  together 
towards  integrated  and  harmonised  actions. 
In  addition,  while  children’s  participation  has 
been  considered  as  a  core  element  for  the 
development  of  trustworthy  AI  for  children, 
there  is  still  a  lack  of  children’s  meaningful 
participation in many initiatives. 

Reaching conclusions through analysis 
and workshops

To  address  these  issues,  three  approaches 
were  used.  The  current  policy  initiatives  for  AI 
and  Children’s  Rights  of  major  international 
organisations  were  reviewed.    In  addition,  the 
scientific evidence of the impact of a selection 
of three AI-based applications on children was 
analysed.  To  complement  these  reviews  and 
analyses,  a  series  of  workshops  with  children, 
researchers and policymakers were held. These 
three  communities  were  deemed  to  be  key 
stakeholders when considering AI and children’s 
rights. The findings from these three approaches 
led  to  the  identification  of  key  requirements, 
methods  and  knowledge  gaps  that  need  to 
be  comprehensively  addressed  in  research 
and  policy  agendas,  to  ensure  that  children’s 
fundamental  rights  are  respected  when  they 
are interacting with AI-based technology.  

The  main  requirements  identified  are  in  line 
with  the  seven  requirements  for  trustworthy 
AI, as detailed by the High-Level Expert Group 
for AI of the European Commission.¹ Moreover, 
methods  that  are  considered  of  use  for 
researchers and policy-makers for engagement 
were  highlighted.  Finally,  knowledge  gaps  that 
need  to  be  addressed  as  priority  in  the  short- 
and medium-term are listed (see page 6).

Topic priorities of experts, policymakers 
and children

It 
is  pertinent  to  understand  how  these 
requirements, methods and knowledge gaps are 
prioritised by different stakeholders, specifically 
children, academia and policymakers. A content 
analysis  of 
the  corresponding  workshops 
indicated that the different groups emphasised 
and  prioritised  different  topics.  While  children 
and policymakers placed emphasis on the impact 
of AI in education, researchers emphasised its 
impact  on  children’s  cognition,  behaviour  and 
development. In addition, inclusion and the fight 
against discrimination appeared to be a topic of 
priority only for policymakers and researchers. 
Interestingly,  transparency  and  explainability 
of  AI  technology  seems  to  be  important  for 
researchers  and  children,  while  policymakers 
gave this the lowest priority of all.

Future steps

A  harmonised  and  coordinated  action  towards 
trustworthy  AI  for  children  would  benefit  from 
transparent communication among stakeholders 
regarding  the  differences  in  the  respective 
agendas  and  priorities.  This  report  details  the 
main outcome of the AI and Rights of the Child 
(AIRoC) research activity. It provides the above-
mentioned  requirements  and  methodological 
tools and the identified knowledge gaps to pilot 
specific  applications  in  real-world  situations 
with children in Europe.

NOTES
¹ https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai

22

 
 
 
Requirements

AI minimisation, valuable purpose and sustainability:  The use of AI technology should be 
considered as a limited resource. Strategic and systemic choices should be made to develop 
AI-based services for children, both at a public and private level.

Transparency, explainability, communication and accountability: These elements should be devel-
oped to inform and empower young citizens and all users of AI technology, preventing mistrust or 
over-trust in AI systems.

Inclusion and non-discrimination:  AI technology should be child-friendly when appropriate and not 
reflect discriminatory biases.

Privacy, data protection and safety:  Children and their carers should be facilitated and empowered to 
control how their personal data are exploited by AI technology.

Integration and respect of children's agency:  Children should be integrated further in conscious 
interactions with AI technology and research should support the construct of children’s agency in this 
context.

Methods

Anticipation, evaluation and monitoring: Frameworks and toolkits that can enable/guide the design 
and evaluation of child-friendly AI systems in the short- and long-term should be created and developed. 
These frameworks should incorporate aspects such as personal data protection and risk assessment.

Multi-stakeholder collaboration: Communication and collaboration between stakeholders should be 
supported to seek balance in the use of AI technology by children and resolve the conflicts between their 
provision, protection and participation rights. Communication and collaboration are key to define common 
goals and to build around a child-friendly AI by design.

Children’s participation:  The cognitive and socio-emotional stage of each child should be taken into 
account while ensuring their fundamental rights (Participation, Provision and Protection) in the design, 
development and use of AI technology. 

Balancing conflicting rights:  Regulation and policies on AI technology should consider the full range of 
children’s rights (Participation, Provision and Protection) and should define the correct degree of compro-
mise between different, sometimes conflicting rights.

Knowledge gaps 

Children’s cognition, development and play: Designers and researchers should systematically study 
the impact of the use of AI technology on children’s cognitive and socio-emotional capacities in different 
contexts and in an inclusive way.

Empowerment through education: Schools should prepare children for a world transformed by AI 
technology and develop adequately their competences and literacy, also supported by AI technology itself 
that can develop competences supporting children’s well-being.

Developmentally appropriate systems and age verification:  AI-based systems targeting children 
should be developed to fit appropriately children’s cognition stage. Age recognition tools should consider 
the large variety of skills, capacities and agency level of children sharing the same age. AI systems should 
adapt to different children needs as well as children's views on how to safeguard their privacy.

33

44

FIGURE 1 - Structure of the report
Source: EC

Structure of the report

Chapter 2
Existing policy initiatives on AI and 
children

Chapter 3
Analysis of the impact of 
selected AI applications on children’s rights

UNCRC

UN General Comment 25

UNICEF

OECD

UNESCO

IEEE

Council of Europe

European Commission

Risks / Opportunity / Challenges

Recommender Systems

Conversational Agents

Robotic systems

Chapter 4
Youth’s perspective: from the workshops 
with children and young people

Chapter 5
Experts’ perspective: from the workshops 
with scientists and policymakers

Preconceptions of AI

Perceptions of the impacts of AI on children’s right

Young people and  their concerns

What do young people propose

Questions for futher directions

* Considering the results from Chapters 2, 3 and 4

    Requirements

  AI minimisation and valuable purpose

  Transparency, explainability, communication and  
  accountability

  Inclusion and non-discrimination

  Privacy, data protection and safety

  Integration and respect of children‘s agency

    Methods

  Anticipations, evaluation, monitoring

  Multi-stakeholder collaboration

  Children’s participation

  Balancing conflicting rights

    Knowledge gaps

  Children’s cognitions, development & play

  Empowering through education

  Developmentally appropriate systems and age verification

  Other research questions to be addressed

Chapter 6
Researchers, policymakers and children: 
A triangulation of perspectives

Chapter 7
Limitations and considerations

Chapter 8
Conclusions

Chapter 9
For deeper consideration:
reflections of invited experts

55

66

CHAPTER 1

1. INTRODUCTION

With  Artificial  Intelligence  (AI)  and  Robotic 
technology  on  the  rise,  an  increasing  number 
of  novel  applications  are  currently  affecting 
–  and  have  the  potential  to  impact  further  – 
children’s well-being and development in terms 
of formal learning and informal play.² The use 
of  AI-based  technology  for  children  provides 
unique  opportunities  in  ways  that  are  novel 
and  have  great  potential  to  support  children’s 
well-being  and  their  development.  However, 
there  is  a  general  concern  that  AI-based 
applications  carry  certain  risks  that  might  not 
sufficiently  support  –  or  might  even  violate  – 
children’s fundamental rights. For example, AI-
based systems that record children’s individual 
preferences  and  provide  personalised  content, 
might present privacy and data protection risks. 

face 

Similarly,  systems  with 
recognition 
modules  might  discriminate  against  specific 
groups of children. These concerns have created 
the need for the introduction of specific policies 
by  governmental  and  non-governmental 
institutions,  with  the  aim  of  mitigating  the 
emerging  risks,  as  well  as  supporting  and 
enlarging  the  potential  opportunities  of  AI-
based technology for children. 

The  protection,  upholding  and  promotion  of 
children’s  rights,  as  they  appear  in  the  United 
Nations’ Convention on the Rights of the Child³ 
have  a  profound  effect  on  children’s  wellbeing 
and 
their  physical,  cognitive,  and  socio-
emotional  development.  For  the  purposes  of 
this  report,  UNICEF’s  definition  for  children’s 
well-being is used, being defined as “children’s 
health and safety, their material security, their 
education and socialisation, and their sense of 
being loved, valued, and included in the families 
and societies in which they are born”.⁴ 

The  term  ‘child  development’  refers  to  the 
growth  and  development:  i.e.  to  the  physical, 

cognitive,  emotional,  and  social  changes  an 
individual experiences from infancy through to 
adolescence (Levin, 2011), which has a dynamic 
nature over time. Indeed, ‘children’s’ knowledge 
of  information  technology  and  AI  may  vary 
substantially over the course of a few years as 
they pass from infancy to adolescence. 

As adolescents, they may have a more advanced 
and  intuitive  understanding  of  some  of  the 
challenges  raised  by  AI  than  their  parents, 
or  even  than  researchers  and  policymakers. 
However, with the current advances in AI-based 
technology,  the  causal  relationship  between 
children’s  rights  and  well-being  is  increasingly 
being  mediated  by  AI-based  applications 
which  may  be  catalytic  for  the  improvement 
of  all  children’s  lives  or  be  a  tool  that  might 
put children at risk. The way that the AI-based 
applications  are  designed,  developed,  and 
deployed requires special attention in order for 
us to ensure the best interests of all children.

While  there  are  already  several  initiatives 
from  governmental  and  non-governmental 
institutions 
that  propose  policy  directions 
towards the upholding of children’s rights in the 
digital  world  some  national  and  international 
policy actions have only recently started to show 
that  AI  brings  specific  risks  and  opportunities 
in  relation  to  children’s  rights,  and  which 
should  be  considered  with  special  attention. 
These  initiatives  indicate  the  importance  of 
understanding the impact of AI-based systems 
on children’s development and well-being, and 
identify  ways  in  which  AI  can  contribute  to 
all  children’s  empowerment,  while  taking  the 
necessary actions to mitigate possible emerging 
risks. 

Scientific evidence about the impact of AI-based 
technologies on children is growing quickly, and it 
has already shaped some initial understanding. 

NOTES
² For the purposes of this report, any human being below the age of 18 is termed as ‘a child’ https://www.unicef.org/child-rights-convention/
convention-text

³ https://www.ohchr.org/en/professionalinterest/pages/crc.aspx

⁴ https://www.unicef-irc.org/publications/pdf/rc7_eng.pdf 

77

CHAPTER 1

However,  the  complexity  of  these  systems  on 
the one hand, and children’s rapid development 
and  adaptive  behaviour  on  the  other  hand, 
render  this  endeavour  particularly  challenging. 
While there is a consensus regarding the need 
for an expansion of such initiatives, further work 
should  be  done  towards  formulating  concrete 
plans  and  guidelines,  as  well  as  directions  for 
their realisation.  

Scope
This  Science  for  Policy  report  is  published  in 
the context of a cross-cutting research activity 
of  the  Joint  Research  Centre  (JRC),  European 
Commission, on the topic of Artificial Intelligence 
and the Rights of the Child (AIRoC). 

The JRC is the European Commission’s science 
and  knowledge  service,  providing  scientific 
evidence  throughout  the  whole  policy  cycle. 
Among the current priorities of the Commission, 
AI has become an area of strategic relevance, 
as  the  European  approach  to  AI  is  based  on 
ensuring  excellence  and  trust,  while  aiming 
to  boost  research  and  industrial  capacity,  and 
at  the  same  time  guaranteeing  fundamental 
rights. 

Considering  that  AI  is  an  integral  part  of 
children’s everyday life, it is essential to adopt a 
children-centric  approach  to  understand  which 
are the specific challenges and means to protect 
children's rights, while seizing the opportunities 
AI offers. 

Within the overall recognition regarding the need 
for concrete, evidence-based actions on AI and 
children’s rights, the JRC AIRoC research activity 
is  framed  under  the  Cybersecurity  Education, 
Awareness and Societal aspects (CEAS) project. 
It aims to explore, and contribute to the current 
knowledge regarding AI and the implications of 
its development and use in relation to children 
and their rights. 

This  project  is  linked  to  the  Human  behaviour 
and  Machine  Intelligence  (HUMAINT)  project 
of  the  JRC,  which  researches  the  impact  of  AI 
on human behaviour, with a focus on decision-
making, as well as cognitive and socio-emotional 
development (Gómez et al., 2021). 

The  AIRoC  research  activity  falls  within  the 
wider activities of the European Commission to 
promote the transformation of Europe into the 
global hub for trustworthy Artificial Intelligence 
(AI).⁵

The  AIRoC  research  activity  takes  a  research-
oriented  perspective  and  seeks  to  identify 
how  the  current  state-of-the-art  of  scientific 
evidence  can  inform  policy  directions.  Even 
though AI-based applications have only recently 
started to be widely used by children, research 
in various areas of AI in relation to children has 
already some evidence to exhibit. 

With  this  research  activity,  we  build  upon  the 
existing scientific results concerning the impact 
of  AI-based  applications;  we  seek  to  examine 
these issues from the perspective of children’s 
rights;  and  we  propose  future  research  and 
policy directions. The expected outcome of the 
research  activity  is  to  draw  recommendations 
for  researchers, 
industry  practitioners  and 
policymakers  on  how  to  protect  and  uphold 
children's rights in an AI context.   

The goal of this report is to review the existing 
initiatives  on  children’s  rights  in  the  context 
of  AI  and  discuss  the  current  state-of  the-art 
on  specific  AI-based  applications  for  children, 
in  order  to  formulate  specific  research  and 
policy-related  challenges  to  be  addressed 
industry  and  policymakers. 
by  academia, 
Directions  that  go  beyond  the  identification  of 
ethical  guidelines  concerning  the  development 
of  methods  for  their  practical  implementation 
are also considered. 

It is outside the scope of this report to provide 
an exhaustive review of policy initiatives and AI 
applications for children. For this reason, certain 
criteria  for  the  selection  of  a  subset  of  these 
have  been  applied  (as  indicated  in  Chapters  2 
and 3).

Technical definitions
For  the  purposes  of  this  report,  a  set  of 
technical  definitions  to  facilitate  the  narrative 
are  provided.  These  working  definitions  are 
addressed to non-specialist audiences.

NOTES
⁵  https://ec.europa.eu/commission/presscorner/detail/en/IP_21_1682

88

CHAPTER 1

Trustworthy AI: concept proposed by the High-
Level  Expert  Group⁸  of  the  EC,  which  consists 
of seven key requirements: human agency and 
oversight;  technical  robustness  and  safety; 
privacy  and  data  governance;  transparency; 
diversity,  non-discrimination  and 
fairness; 
societal  and  environmental  well-being;  and 
accountability.

Children’s  rights:  children  are  entitled  to  all 
human rights – as anyone else. In addition, they 
are granted particular rights taking into account 
specificities, vulnerabilities and age-appropriate 
needs.⁹ Children’s rights consider the necessity 
of  the  development  of  the  child,  and  they 
imply  the  need  to  protect  children  and  grant 
assistance  to  them,  adapted  to  their  age  and 
degree of maturity. These particular rights are 
enshrined in the United Nations Convention on 
the Rights of the Child (UNCRC): children’s rights 
are  thus  human  rights  specifically  adapted  to 
the child. Children’s rights are duly reflected in 
the  Charter  of  Fundamental  Rights  of  the  EU 
and EU secondary law. 

Definitions  of  additional  terms  will  be  given, 
where necessary, throughout the document.

AI system: the definition of an AI system as put 
forward in the AI Act proposal⁶  has been adopted. 
This considers an artificial intelligence system (AI 
system) as software that is developed with one 
or more of the techniques and approaches listed 
in Annex I of the AI Act proposal. For a given set 
of human-defined objectives, such software can 
generate  outputs  such  as  content,  predictions, 
recommendations,  or  decisions  influencing  the 
environments they interact with. Annex I of the 
AI  Act  contains  a  list  of  artificial  intelligence 
techniques  and  approaches  including  machine 
learning,  knowledge-based  approaches  and 
statistical models. In this report, AI systems that 
are embodied in robots are also considered, as 
explained below.

AI  actors:  AI  actors  are  those  who  play  an 
active role in the AI system lifecycle, including 
organisations  and  individuals  that  produce  or 
use AI systems.

AI  stakeholders:  stakeholders  encompass  all 
organisations  and  individuals  involved  in,  or 
affected by, AI systems, directly or indirectly. AI 
actors are a subset of stakeholders.

AI  system  adaptation: the process by which 
an AI system learns to change its behaviour by 
analysing the current state of the environment 
and  how  the  environment  is  affected  by  its 
previous actions. For example, adaptive learning 
platforms have the potential to provide adaptive 
personalised  learning  experiences  to  address 
each user’s needs.

(i) 

AI system lifecycle: this definition is adopted 
from  the  OECD’s  Recommendation  of  the 
Council  on  Artificial  Intelligence.⁷  AI  system 
lifecycle  phases 
‘design,  data 
involve: 
and  models’,  which  are  a  context-dependent 
sequence  encompassing  planning  and  design, 
data  collection  and  processing,  as  well  as 
model building; (ii) ‘verification and validation’; 
‘operation  and 
(iii) 
monitoring’.  These  phases  often  take  place  in 
an  iterative  manner  and  are  not  necessarily 
sequential. 

‘deployment’;  and 

(iv) 

NOTES
⁶ https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52021PC0206

⁷ https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0449

⁸ https://digital-strategy.ec.europa.eu/en/policies/expert-group-ai

⁹ https://www.ohchr.org/en/professionalinterest/pages/crc.aspx 

99

1010

CHAPTER 2

2. EXISTING POLICY INITIATIVES ON AI AND 
CHILDREN

This section presents and discusses a selection of existing international policy initiatives 

on  AI  and  children’s  rights.  The  analysis  of  initiatives  at  a  national  level,  and  policy 
initiatives  from  industry  and  NGOs  is  beyond  the  scope  of  this  report.  Some  of  the 
initiatives have not been developed specifically for children, but they have been used as a 
basis for further consideration of children’s rights.

Overall, the initiatives examined are aligned to a certain extent regarding the risks and the 
opportunities of AI for children, but they differ in terms of goals and priorities. For instance, 
OECD  initiatives  focus  mainly  on  education;  UNICEF  takes  a  more  generic  approach  to 
children's rights; and the EC policy initiatives address mainly AI in education while taking a 
risk-based approach with the AI Act. The IEEE takes a more practical perspective and actively 
includes  best  practices  from  industry  to  draw  conclusions  about  the  methodologies  and 
techniques that are used for the upholding of children’s rights in the design, development, 
and deployment of specific AI-based products for children.

In  all  initiatives,  we  observe  that  children’s  participation  is  recognised  as  a  fundamental 
requirement. However, some organisations, such as UNICEF, consider children’s participation 
as  a  priority.  In  addition,  all  initiatives  recognise  the  need  for  practical,  actionable 
methodologies that would pilot policy guidance for AI and children’s rights and eventually 
support AI development and use.

While  all  initiatives  recognise  the  importance  of  scientific  evidence  in  policy  decision 
making, they do not explicitly discuss how science has informed their policy decisions or the 
methodological challenges involved in the implementation of research in policy related to AI 
for children. 

This connection between policymaking and scientific research on AI and child’s rights is the 
focus of this report.

1111

CHAPTER 2

2.1 The UN Convention on the Rights 
of the Child

is  a 

We take as a starting point the United Nations 
Convention on the Rights of the Child (UNCRC), 
which 
international 
legally-binding 
agreement  setting  out  the  civil,  political, 
economic,  social  and  cultural  rights  of  every 
child,  regardless  of  their  race,  religion  or 
abilities.¹⁰  The convention has 54 articles, and 
it has become the most widely ratified human 
rights  treaty  in  history,  helping  transform 
children’s  lives  around  the  world.  The  UNCRC 
has been ratified by all EU Member States and 
neighbouring countries. 

Technological advances have radically changed 
the  world  in  which  children  develop,  play  and 
learn.  Even  though  the  rights  and  principles 
established in the CRC are also applicable to the 
digital world, and UN CRC General Comment 25 
provides concrete guidance on this, governments, 
businesses  and  communities  systematically 
take specific actions to ensure that every child 
has every right in the technologically changing 
environment children face.

2.2 The UN General Comment 25

Given  the  evolving  and  expanding  digital 
environment and the opportunities and the risks 
that the digital world brings for children, the UN 
adopted  the  Comment  25  of  the  rights  of  the 
child  in  the  digital  environment    after  a  large 
consultation  with  children  and  all  interested 
parties.¹¹ 

This  comment  considers  children’s  possible 
interactions  within  digital  networks,  content, 
services  and  applications,  connected  devices 
and  environments,  virtual  and  augmented 
reality, artificial intelligence, robotics, automated 
systems,  algorithms  and  data  analytics, 
biometrics  and  implant  technology  and  invites 
governments to take the appropriate actions in 
order to mitigate the possible risks and ensure 
that the new opportunities for the realisation of 

children’s rights are accessible to all children in 
the digital world. 

This  Comment  elaborates  on  the  four  basic 
principles of the Rights of the child namely 

▶ Non-discrimination; 

▶ Best interest of the child;

▶ Right to life survival and development; and 

▶ Respect for the views of the child.

It  proposes  a  set  of  general  measures  of 
implementation  by  States  parties,  such  as  the 
review  and  possible  adoptions,  amendments 
and  updates  in  the  national  legislation  in  line 
with  the  international  human  rights  standards 
and  the  implementation  of  national  policies 
relating  to  children’s  rights  to  specifically 
address the evolving digital environments. 

The  Comment  25  invites  governments      to 
implement regulation, industry codes, standards 
and  plans  to  provide  children  with  beneficial 
opportunities  from  engaging  with  the  digital 
environment in a safe manner. After a detailed 
consideration  of  how  the  rights  of  the  child 
should be viewed in the digital environment, the 
document  indicates  that  the  cross-border  and 
transnational nature of the digital environment 
international  and  regional 
requires  strong 
cooperation  to  ensure  that  all  stakeholders, 
including  businesses  and  other  actors, 
effectively  respect,  protect  and  fulfil  children’s 
rights in relation to the digital environment.

This  General  Comment  25  acknowledges  that 
in  developed  countries,  navigating  the  digital 
world is part of children’s everyday life, and the 
global  community  has  started  to  understand 
the  challenges  and  the  opportunities  digital 
environments  bring  to  children.  However,  it 
indicates that special attention should be paid 
to AI-based technologies that have been shown 
to  change  fundamentally  children’s  interaction 
with the digital and physical world.

NOTES
¹⁰ https://www.ohchr.org/en/professionalinterest/pages/crc.aspx

¹¹ https://www.ohchr.org/EN/HRBodies/CRC/Pages/GCChildrensRightsRelationDigitalEnvironment.aspx

1212

CHAPTER 2

The  current  scientific  evidence  about  the 
impact  of  AI-based  applications  on  a  child’s 
development and the kind of opportunities and 
risks they bring is still limited. Even less explored 
is how the rights of the child, as defined in 
the UNCRC and the General Comment 25, 
can be realised in the AI context.

of  this  consultation  was  the  observation  that 
there is an adult-based bias in the current AI-
based technologies.

In addition, a review of 20 National Strategies 
on AI revealed that most make only a cursory 
mention of children and their specific needs.¹⁴ 

2.3 The UNICEF initiative on AI and 
children’s rights
In 2019, UNICEF and the Government of Finland 
launched a two-year project to focus specifically 
on  children’s  rights  in  the  context  of  artificial 
intelligence.¹²  

This  project  aimed  to  understand  better  how 
AI-based systems will affect children and their 
rights,  and  what  kind  of  actions  are  needed 
to  protect,  provide  for,  and  empower  children 
worldwide. UNICEF considered that children have 
difficulties  understanding  AI-based  systems 
while  having  no  opportunities  to  express  their 
opinion in decision-making processes. 

For this reason, the inclusion of children in the 
process  of  the  development  of  the  relevant 
Policy Guidelines was a priority. In this context, 
UNICEF  invited  a  wide  range  of  stakeholders, 
including  children,  governmental  and  non-
governmental  organisations,  businesses  and 
experts from the field of AI and child’s rights to 
contribute to the development of a set of Policy 
Guidelines for AI and children’s rights. 

The  organisation  of  a  series  of  global  and 
regional,  workshops  with  experts  and  children 
was 
instrumental  to  the  development  of 
those  guidelines.¹³  UNICEF  involved  245 
adolescents  from  Brazil,  Chile,  South  Africa, 
Sweden  and  the  United  States,  in  a  global 
consultation. 

The children had the opportunity to discuss their 
experiences  with  AI  and  how  they  understand 
the  risks  and  the  opportunities  it  presents  in 
their lives. One of the most prevalent outcomes 

Moreover,  little  attention  is  explicitly  being 
given to safeguarding the rights of children in 
an  algorithmic-oriented  economy  and  society; 
while measures for the preparation of children 
to  live  in  an  AI  world  and  develop  basic  AI 
literacy skills need to be significantly expanded. 
This  report  invites  policymakers  to  consider 
the  re-prioritisation  of  children’s  rights  in  the 
National AI policies to be of critical importance.
Based  on  the  above-mentioned  activities, 
UNICEF published a report on Policy Guidance 
for  AI  and  Children  in  November  2021.¹⁵  It 
proposed nine requirements and corresponding 
recommendations  for  the  development  and 
deployment of child-centred AI, namely:

1.  Support children’s development and well-be 

  ing; 

2.  Ensure inclusion of and for children;

3.  Prioritise fairness and non-discrimination for  

  children;

4.  Protect children’s data and privacy;

5.  Ensure safety for children;

6.  Provide  transparency,  explainability  and  ac     

  countability for children;

7. Empower governments and businesses with  

 knowledge of AI and children’s rights;

8.  Prepare  children  for  present  and  future  de 

 velopments in AI;

9.  Create an enabling environment for child-cen 

  tred AI;

NOTES
¹² https://www.unicef.org/globalinsight/featured-projects/ai-children 

¹³ https://www.unicef.org/globalinsight/stories/adolescent-perspectives-artificial-intelligence 

¹⁴ https://www.unicef.org/globalinsight/media/1156/file

¹⁵ https://www.unicef.org/globalinsight/media/2356/file/UNICEF-Global-Insight-policy-guidance-AI-children-2.0-2021.pdf 

1313

CHAPTER 2

The  report 
indicates  that  a  foundational 
framework  for  the  implications  of  the  above-
mentioned requirements prioritises:

protecting  children  from  risk  and  promoting 
the  opportunities  and  benefits  that  the  digital 
environment can provide. 

▶ Protection = do no harm;

▶ Provision = do good;

▶ Participation = include all children.

The  report  underlines  that  when  applying  this 
foundation  to  AI  policies,  systems  design, 
development  and  deployment,  regardless  of 
regulatory  frameworks,  children  are  always 
entitled  to  their  rights.  Finally,  this  guidance 
indicates  specific  opportunities  and  risks  that 
have  been  correlated  with  specific  AI-based 
technologies, such as chatbots, recommendation 
systems,  robots  and  automated  decision-
making  systems  and  different  AI  techniques, 
such as natural language processing, computer 
vision, and reinforcement learning.

For  the  development  of  this  report,  UNICEF 
invited  a  group  of  governmental  and  non-
governmental  organisations  and  businesses 
to  pilot  the  proposed  guidelines  with  real 
applications  and  policies,¹⁶  and  share  their 
findings  (the  JRC  participated  as  one  of  the 
pilot partners in collaboration with the HONDA 
Research Institute, Japan).¹⁷  

The  piloting  of  the  Policy  Guidance  for  AI  and 
Children’s Rights contributed to the improvement 
of  the  guidelines  and  entailed  implementation 
examples  for  the  design,  development  and 
deployment of child-centred AI.

2.4 OECD related initiatives

In  May  2021,  the  OECD  published  a  revision 
of  the  “Recommendation  on  the  Protection  of 
Children  Online”  in  view  of  the  technological, 
legal  and  policy  advances,  which  was 
renamed  as  “Recommendation  on  Children 
in  the  Digital  Environment”.¹⁸  The  goal  of  this 
recommendation  is  to  find  a  balance  between 

These 
recommendations  are  based  on 
the  OECD’s  five  principles  for  responsible 
stewardship  of  trustworthy  AI,  as  referred  to 
in  the  OECD’s  Legal  Instrument  of  the  Council 
on  Artificial  Intelligence,¹⁹    and  call  on  all  the 
Members  and  Non-members,  as  well  as  on 
all  the  actors  involved  to  engage  proactively 
in  responsible  stewardship  of  trustworthy  AI 
by  integrating  the  following  principles  for  the 
development of AI systems:

▶ Inclusive  growth,  sustainable  development  

 and well-being;

▶ Human-centred values and fairness;

▶ Transparency and explainability;

▶ Robustness, security and safety;

▶ Accountability.

for 

The  above-mentioned  principles 
the 
development  of  trustworthy  AI  should  be 
taken  into  consideration  with  the  overarching 
child-specific  framework  as  it  appears  in  the 
Recommendations  on  Children  in  the  Digital 
Environment,  and  which  emphasises  to  the 
following points: 

▶  Review,  develop,  and  amend  as  appropriate, 
laws that directly or indirectly affect children 
in the digital environment;

▶  Promote  digital  literacy  as  an  essential  tool 

for meeting the needs of children; 

▶ Adopt evidence-based policies;

▶ Promote the adoption of measures that pro-
vide  for  age-appropriate  child  safety  by  de-
sign.

NOTES
¹⁶ https://www.unicef.org/globalinsight/policy-guidance-ai-children-pilot-testing-and-case-studies 

¹⁷ https://www.unicef.org/globalinsight/media/2206/file  

¹⁸ https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0389

¹⁹ https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0449 

1414

CHAPTER 2

The OECD has formulated clear recommendations 
for  the  development  of  trustworthy  AI  and 
for  actionable  points  for  the  best  interests  of 
children regarding their navigation of the Digital 
Environment. Still, it draws special attention to 
the  role  of  children’s  AI  literacy  and  the  role 
of  AI  in  education  with  the  “Trustworthy  AI  in 
education: promises and challenges” report.²⁰  

One  of  the  current  challenges,  though,  is  the 
limited  evidence  about  the  effectiveness  of 
many  AI  solutions  in  education,  and  the  need 
for  new  forms  of  innovative  school  networks 
since  the  traditional  ways  of  assessing  the 
effectiveness  of  educational  interventions  (e.g. 
randomised control trials) may be too slow in a 
rapidly  evolving  technological  context.  Another 
important  policy  consideration  is  questioning 
how  governments  can  work  with  stakeholders 
to shape AI in education, in order to help prepare 
for the transformation of the world of work and 
society,  and  to  define  what  appropriate  data 
sharing means. 

With an extensive analysis of various AI-based 
applications,  the  OECD  proceeded  with  the 
exploration of the current state-of-the-art and 
recommendations  for  AI  in  education  in  its 
“OECD Digital Education Outlook 2021: Pushing 
the Frontiers with AI, Blockchain and Robots”.²¹  

One of the main conclusions of this report is that 
effective  use  of  robots,  classware,  predictive 
analytics  and  similar  technology  will  require 
reinventing the role of teachers and will require 
international and cross-sectoral collaborations. 

Finally, the OECD recognises children’s AI literacy 
as an essential part of their education and their 
socio-emotional  development,²²  as  indicated 
in  the  first  AI  principle  of  inclusive  growth, 
sustainable development and well-being.²³

2.5 UNESCO related initiatives

The  United  Nations  Educational,  Scientific  and 
(UNESCO)  recognises 
Cultural  Organisation 
that  AI  can  be  a  powerful  tool  to  address 
current  global  challenges  and  to  advance 
human capabilities. However, it also recognises 
that,  along  with  multiple  advantages,  these 
technologies  generate  downside  risks  and 
challenges,  derived  from  the  malicious  use 
of  technology  or  deepening  inequalities  and 
divides.  Moreover,  it  highlights  the  need  for 
international  and  national  policies,  as  well  as 
regulatory  frameworks  to  ensure  that  these 
emerging  technologies  benefit  humanity  as  a 
whole.²⁴

To 
this  end,  UNESCO  published  drafted 
recommendations on the ethics of AI in 2019, 
to provide a basis to make AI systems work for 
the  good  of  humanity,  individuals,  societies, 
environment  and  ecosystems,  and  to  prevent 
harm.²⁵  In  November  2021,  all  193  member 
states  of  UNESCO  unanimously  adopted  the 
recommendations on the ethics of AI, which aim 
to  realise  the  advantages  of  technology  while 
reducing  the  risks  to  human  rights  associated 
with its use.

Regarding the AI-related activities for education, 
UNESCO  adopted  the  UN  General  Comment 
25  on  Children’s  Rights  in  Relation  to  the 
Digital Environment,²⁶  and following a similar 
approach  to  OECD,  UNESCO  categorised  AI-
based applications for education into two large 
categories that contribute to the improvement 
of learning and equity for all children, namely:

1.  AI  to  promote  personalisation  and  better 
learning outcomes; and

2.  Data  analytics  in  Education  Management 
Information Systems (EMIS) and the evolution to 
Learning Management Systems (LMS)  

NOTES
²⁰ https://www.oecd-ilibrary.org/docserver/a6c90fa9-en.pdff?expires=1633947478&id=id&accname=guest&checksum=  
   F2EFBB42D1F4EC64A20E7B827FC3DF26

²¹ https://www.oecd-ilibrary.org/education/oecd-digital-education-outlook-2021_589b283f-en

²² https://www.oecd.org/education/ceri/social-emotional-skills-study/beyond-academic-learning-92a11084-en.htm 

²³ https://oecd.ai/en/ai-principles

²⁴ https://en.unesco.org/artificial-intelligence/ethics 

²⁵ https://unesdoc.unesco.org/ark:/48223/pf0000377897 

²⁶ https://en.unesco.org/news/unesco-welcomes-new-international-instrument-childrens-rights-relation-digital-environment

1515

CHAPTER 2

In its report “Artificial Intelligence in Education: 
Challenges  and  Opportunities  for  Sustainable 
Development”,²⁷  UNESCO  identified  a  number 
of recommendations that can support relevant 
stakeholders to promote the following actions:

▶ Develop a comprehensive view of public poli-

cy on AI for sustainable development;

▶  Ensure  inclusion  and  equity  for  AI  in  educa-

tion;

▶ Prepare teachers for AI-powered education;

▶ Develop quality and inclusive data systems;

▶ Enhance research on AI in education;

▶ Deal with ethics and transparency in data col-

lection, use and dissemination.

To  facilitate  the  design  of  concrete  actionable 
points by policymakers, UNESCO published the 
“AI  and  education:  Guidance  for  policymakers” 
identifies  specific  areas  for 
report,  which 
action.²⁸ 
the  children’s 
participation  has  been  considered  essential 
in  this  process,  UNESCO  in  collaboration  with 
Ericson,  which  functions  as  a  repository  for 
educators  and  curriculum  developers,  aim  to 
prepare  and  exchange  learning  resources.²⁹  

In  parallel,  since 

2.6 IEEE related initiatives

The 
Institute  of  Electrical  and  Electronics 
Engineers  (IEEE)  is  the  largest  association  of 
technical  professionals,  aiming  to  advance 
electrical and electronic engineering, computer 
engineering  and  similar  fields.  It  has  been 
particularly  active  in  prioritising  the  emerging 

ethical considerations in the creation of AI and 
autonomous systems.³⁰

In 2016, it launched the IEEE Global Initiative on 
Ethics  of  Autonomous  and  Intelligent  Systems 
to  ensure  every  stakeholder  involved  in  the 
design  and  development  of  autonomous  and 
intelligent  systems  is  educated,  trained,  and 
empowered to prioritise ethical considerations, 
so that these technologies are advanced for the 
benefit of humanity.³¹  

Since then, IEEE has developed a set of guidelines 
for  Ethically  Aligned  Design  by  prioritising 
human  well-being,³²  as  well  as  practices  for 
assessing  the  impact  of  autonomous  and 
intelligent systems on human well-being.³³ 

In  this  context,  the  IEEE  has  showed  a  special 
interest  in  better  understanding  the  impact 
of  AI  on  children’s  lives,  by  first  considering 
parents’ perceptions regarding the use of AI by 
their  children  with  the  “Generation  AI:  A  Study 
of Millennial Parents of Generation Alpha Kids”. 
This study reports on the results of 600 parents 
who  were  surveyed;  the  survey  illuminated 
how  millennial  parents  with  Generation  Alpha 
children  (seven  years-old  or  younger)  think 
growing  up  interacting  with  AI  technology  will 
impact the lives of their children.³⁴  The results 
indicate that millennial parents have a tendency 
to accept AI as an integral part of their children’s 
lives  including  their  education  and  their  social 
life. 

Since  then  and  taking  a  practical  perspective, 
the  IEEE  has  initiated  a  series  of  projects  on 
AI  in  relation  to  children  in  the  context  of  its 
Standards  Association.³⁵  The  global  initiative 
on  the  Ethics  of  Autonomous  and  Intelligent 
Systems was integrated into the IEEE Standards 
Association and seeks to identify methodologies 

NOTES
²⁷ https://en.unesco.org/artificial-intelligence/education

²⁸ https://unesdoc.unesco.org/ark:/48223/pf0000376709

²⁹ https://en.unesco.org/news/unesco-and-ericsson-launch-new-portal-teaching-ai-students 

³⁰ https://standards.ieee.org/content/ieee-standards/en/news/2016/ethically_aligned_design.html 

³¹ https://standards.ieee.org/content/ieee-standards/en/industry-connections/ec/autonomous-systems.html 

³² https://exploreaiethics.com/guidelines/ethically-aligned-design-v2/ 

³³ https://doi.org/10.1109/IEEESTD.2020.9084219

³⁴ https://www.ieee.org/about/news/2017/ieee-unveils-generation-ai.html 

³⁵ https://standards.ieee.org

1616

CHAPTER 2

child-friendly justice and children’s rights in the 
digital  environment.  An  ad  hoc  Committee  for 
the Rights of the Child (CAHENF) was created in 
the same period and tasked by the Committee 
of  Ministers  to  oversee  the  implementation  of 
the strategy. 

Specifically,  on  the  theme  of  AI  technology 
and  children’s  rights,  we  can  highlight  the 
“Guidelines  to  respect,  protect  and  fulfil  the 
rights  of  the  child  in  the  digital  environment 
-  Recommendation  CM/Rec(2018)7  of  the 
Committee  of  Ministers”. 
Its  purpose  and 
scope  are  to  provide  assistance  to  relevant 
stakeholders in the implementation of the rights 
enshrined in international and European human 
rights conventions and standards, in the light of 
the case law of the European Court of Human 
Rights. The guidelines rely on five fundamental 
principles and rights: 

1. Best interests of the child;

2. Evolving capacities of the child;

3. Right to non-discrimination;

4. Right to be heard;

5. Duty to engage other stakeholders.

The  guidelines  developed  seven  operational 
principles  and  measures  to  respect,  protect 
and  fulfil  the  rights  of  the  child  in  the  digital 
environment, namely:

1. Access to the digital environment;

2. Right to freedom of expression and informa-

tion;

3. Participation, right to engage in play and right 

to assembly and association;

4. Privacy and data protection;

5. Right to education;

6. The right to protection and safety;

to  proceed  from  ethical  principles  to  practical 
developments.³⁶ 

the 

initiatives  of 

IEEE  Standard 
Further 
Association  focus  on  the  collection  of  industry 
best practices that uphold children’s rights from 
the  design  stage  of  their  products  onwards. 
For  example,  the  initiative  of  “Children's  Data 
Governance  Applied  Case  Studies”  presents 
a  collection  of  case  studies  that  demonstrate 
the connection of ethical principles in relations 
to  children’s  rights  to  best  practices  to 
ensure  trustworthy  and  positive  online/offline 
experiences for children.³⁷ 

IEEE  Standards 
In  September  2021,  the 
Association  published  the  “Draft  Standard  for 
Age-Appropriate  Digital  Services  Framework  – 
Based  on  the  5Rights  Principles  for  Children” 
(P2089/D4).³⁸  Following  the  growing  desire 
of  companies  to  develop  digital  and  AI-based 
products for children, this standard establishes 
a  set  of  processes  by  which  organisations 
seek  to  make  their  services  Age  Appropriate. 
This  standard  reflects  children’s  existing  rights 
as  they  appear  under  the  UNCRC  by  setting 
processes through the life cycle of development, 
delivery and distribution of the product.

2.7 Council of Europe related 
initiatives
The  Council  of  Europe  questioned  and  led 
policymaking  initiatives  on  the  impact  of  AI 
technology  on  human  rights,  early  in  the  last 
decade. Moreover, since launching the Program 
“Building  a  Europe  for  and  with  Children”  in 
2006,  the  Council  of  Europe  has  implemented 
strategies over a series of policy cycles to guide 
its  work  on  strengthening  the  protection  of 
children’s rights in Europe. In 2016, it published 
the Strategy for the Rights of the Child (2016-
2021), adopted by the Committee of Ministers 
of the Council of Europe. 

The strategy covers priority areas to guarantee 
all children’s rights, namely in the areas of equal 
opportunities, participation, a violence-free life, 

NOTES
³⁶ https://standards.ieee.org/industry-connections/ec/autonomous-systems.html 

³⁷ https://standards.ieee.org/initiatives/artificial-intelligence-systems/childrens-data-governance.html

³⁸ https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=COM%3A2018%3A237%3AFIN

1717

CHAPTER 2

7. Remedies.

The  document  concludes  by  insisting  on  the 
need  for  cooperation  and  coordination  at  the 
national and international level. To support the 
implementation  of  the  recommendations  CM/
Rec(2018)7, the Council of Europe published a 
“Handbook for policymakers on the rights of the 
child in the digital environment”. 

Regarding  the  participation  of  young  people 
in  the  policymaking  process,  the  Council  of 
Europe  published  the  “Declaration  on  Youth 
Participation  in  AI  Governance”  at  the  end  of 
2020,  presenting  the  outcome  of  the  work 
done  by  the  young  people  participating  in  the 
seminar on Artificial Intelligence: How Can Youth 
Take Part? organised by the Youth Department 
of  the  Council  of  Europe,  between  23  and  27 
November 2020. 

This  is  positioned  within  the  wider  framework 
of  the  Council  of  Europe’s  youth  policy  and 
initiatives that link AI and democratic citizenship. 
The declaration explores issues, challenges and 
roles that stakeholders can play to secure and 
enable  the  participation  of  young  people  in  AI 
governance processes at all levels. It highlights 
five important considerations: 

1.  The  exclusionary  consequences  of  the 
fast-paced development of AI technolo-
gies  driven  by  the  private  sector  that  leave 
many  stakeholders,  especially  youth  actors 
and  human  rights  activists,  but  also  policy-
makers,  behind,  and  result  in  normatively 
questionable  and  ineffective  self-regulation 
in the private sector.

2.  The  need  to  develop  and  ensure  legal 
safeguards, both by international organisa-
tions  (through  existing  or  new  legal  instru-
ments) and by national governments respon-
sible for securing and implementing them at 
national level.

3.  The  absence  of  youth  in  the  emerging 
AI governance processes as a denial of the 
right  to  participate  in  democratic  processes 
which impedes a whole sector from co-shap-
ing the discourse about development, assess-

ment,  implementation  and  regulation  of  AI 
technologies.

4. The imperative to respect ethical princi-
ples which must be at the core of all AI de-
velopments  and  deployment  (transparency, 
justice and fairness, responsibility, safety and 
security, privacy).

5. The need to assess the value of AI tech-
nologies on the impact of their consequenc-
es and benefits on individuals and society. Not 
all social and economic problems need to be 
anchored in technological solutions.

Finally, we can point to the activities of the Ad 
Hoc Committee on Artificial Intelligence (CAHAI). 
From  September  2019  to  December  2021, 
and  on  the  basis  of  broad  multi-stakeholder 
consultations,  it  examined  the  feasibility  and 
potential elements of a legal framework for the 
development, design and application of artificial 
intelligence,  based  on  Council  of  Europe’s 
standards on human rights, democracy and the 
rule of law. 

Among the outcomes of this initiative, we can 
highlight the data visualisation of AI initiatives 
tool,  which  brings  together  more  than  500 
documents  related  to  artificial  intelligence, 
coming  from  national  authorities,  the  private 
sector,  international  organisations  or  multi-
stakeholder  initiatives.  However,  we  also  note 
that  the  words,  ‘child’,  ‘children’  or  ‘youth’  are 
not among the concepts listed as frequent.

2.8  European  Commission  related 
initiatives

2.8.1 The European strategy on the rights 
of the child

The  Charter  of  Fundamental  Rights  of  the 
European Union, in particular Art.24, guarantees 
the protection of the rights of the child by the 
EU institutions and by EU countries when they 
implement EU law.³⁹ Regarding the intersection 
of  human  fundamental  rights  and  AI,  the 
European Union Agency for Fundamental Rights 

NOTES
³⁹ https://www.europarl.europa.eu/charter/pdf/text_en.pdf

1818

CHAPTER 2

TABLE 1
Key actions proposed by the “EU strategy on the rights of the child” regarding the support of children’s 
rights in the digital and information society [adopted by the EU Strategy on the Rights of the Child]
Source: EC

 Actor

Key actions

By the European 
Commission:

Adopt an updated Better Internet for Kids Strategy in 2022;

Create and facilitate a child-led process aimed at developing a set of principles to 
be promoted and adhered to by the industry;

Promote the development and use of accessible ICT and assistive technologies for 
children with disabilities, such as speech recognition, closed captioning and others, 
including in Commission’s conferences and events; 

Ensure the full implementation of the European Accessibility Act;

Step up the fight against all forms of online child sexual abuse, such as by 
proposing the necessary legislation, including obligations for relevant online service 
providers to detect and report known child sexual abuse material online.

The European
Commission invites 
the Member States to:

Ensure effective equal access to digital tools and high-speed Internet connection, 
digital literacy, accessible online educational material and education tools etc. for 
all children; 

Support the development of children’s basic digital competences, through the 
Digital Competence Framework for citizens;

Support media literacy actions as part of education, to develop children’s ability to 
evaluate critically online content, and detect disinformation and abusive material;

Support and promote the work of the EU co-funded Safer Internet Centres, and 
support child helplines and hotlines in developing online avenues for 
communication;

Encourage children’s and especially girls’ participation in science, technology, 
engineering and mathematics (STEM) studies and dismantle gender stereotypes in 
this field to ensure equal opportunities in the digital labour market.

The European 
Commission invites 
ICT companies to:

Ensure that children’s rights, including privacy, personal data protection, and 
access to age-appropriate content, are included in digital products and 
services by design and by default, including for children with disabilities; 

Equip children and parents with adequate tools to control their screen time 
and behaviour, and protect them from the effects of overuse and addiction to 
online products; 

Strengthen measures to help tackle harmful content and inappropriate 
commercial communication, such as through easy-to-use reporting and 
blocking channels or effective age-verification tools; 

Continue their efforts to detect, report and remove illegal online content, 
including child sexual abuse from their platforms and services, to the extent 
that these practices are lawful.

1919

CHAPTER 2

(FRA)  analysed  different  risks  in  the  use  of 
AI  from  a  fundamental  rights  perspective, 
and  acknowledges  the  lack  of  awareness  of 
stakeholders regarding the rights of the child in 
the context of AI.⁴⁰

With  the  same  report,  FRA  emphasises  the 
need  for  mainstreaming  children’s  rights  in  AI 
policies. Guided by the principles set out in the 
UN Convention on the Rights of the Child,⁴¹  the 
Commission  published  the  European  Strategy 
for  the  Rights  of  the  Child⁴²    to  address 
persisting and emerging challenges around six 
thematic areas, and to propose concrete actions 
to protect, promote and fulfil children’s rights in 
today’s context. 

The proposed thematic areas, which are based 
on the EU priorities for the coming years, relate 
to  action  points  that  should  be  taken  into 
consideration by the European Commission and 
by  the  Member  States.  Among  the  thematic 
areas that the Strategy proposes is the “Digital 
and  information  society:  an  EU  where  children 
can safely navigate the digital environment and 
harness its opportunities”. 

To  this  end,  the  Strategy  identifies  a  set  of 
initiatives  by  the  EU  that  have  already  been 
adopted,  mainly  towards  children’s  online 
safety and protection, as well as the new Digital 
Education  Action  Plan  (2021-2027)  which 
promotes  digital  literacy  and  puts  education 
and training at the heart of this effort.⁴³  Lastly, 
it highlights the impact that AI has and will have 
on children and their rights, such as in the fields 
of  education,  leisure,  healthcare  provision.  It 
also calls for a more effective fight against Child 
Sexual Abuse.⁴⁴ 

Based on the above-mentioned thematic areas, 
and  going  one  step  further,  the  European 

Commission  has  proposed  certain  key  actions 
that  should  be  considered  in  the  future  for 
upholding  children’s  rights  in  the  digital  and 
information society. 

Table  1  shows  the  invitation  of  the  European 
Commission  of  certain  actors  (the  European 
Commission,  the  Member  States,  and  the  ICT 
companies) to consider a set of key actions (see 
Table 1).

2.8.2 Artificial intelligence policies at the 
European Commission

The  General  Data  Protection  Regulation 
(GDPR)  consists of a harmonised set of data 
protection rules across the EU to protect citizens’ 
fundamental  rights  in  today's  digital  society.⁴⁵ 
One  of  the  main  goals  of  the  regulation  is 
for  citizens  to  be  in  control  of  their  personal 
data  and  to  ensure  the  protection  of  their 
fundamental rights. 

The  GDPR  applies  to  AI  systems  that  process 
personal  data.  It  contains  specific  provisions 
that refer to profiling and to automated decision 
systems,  many  of  which  are  based  on  AI.  The 
GDPR  recognises  that  children  need  special 
protection  regarding  their  personal  data,  and 
it  contains  provisions  aimed  at  securing  the 
processing  of  their  data  and  to  ensure  that 
children understand and can exercise their data 
protection rights. 

In  the  context  of  the  European  strategy  for 
data,⁴⁶  the European Commission has proposed 
the Regulation on European data governance⁴⁷  
and the Data Act⁴⁸ that aim to unlock the re-
use potential of different types of data, and to 
create common European data spaces.

NOTES
³⁹ https://www.europarl.europa.eu/charter/pdf/text_en.pdf

⁴⁰ https://fra.europa.eu/en/publication/2020/artificial-intelligence-and-fundamental-rights 

⁴¹ https://www.unicef.org.au/upload/unicef/media/unicef-simplified-convention-child-rights.pdf

⁴² https://eur-lex.europa.eu/resource.html?uri=cellar:e769a102-8d88-11eb-b85c-01aa75ed71a1.0002.02/DOC_1&format=PDF

⁴³ https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52020DC0624

⁴⁴ https://ec.europa.eu/home-affairs/system/files/2020-07/20200724_com-2020-607-commission-communication_en.pdf 

⁴⁵ https://gdpr.eu/what-is-gdpr/ 

⁴⁶ https://digital-strategy.ec.europa.eu/en/policies/strategy-data

2020

CHAPTER 2

In  relation  specifically  to  AI,  the  European 
Commission  has  launched  a  set  of  initiatives 
related to Artificial Intelligence since 2018, that 
we summarise as follows: 

The  launch  of  the  European  AI  Strategy 
in  2018,  with  a  Communication  on  Artificial 
Intelligence  for  Europe  and  the  Coordinated 
Plan on AI,⁴⁹  prepared with Member States to 
foster the development and use of AI in Europe. 
This plan proposes joint actions for a strategic 
cooperation  between  Member  States,  Norway, 
Switzerland and the Commission on three main 
pillars:  to  ensure  technological  developments 
and uptake, to prepare for the socio-economic 
changes  brought  by  AI,  and  to  lay  out  an 
appropriate ethical and legal framework.

The work of the High-Level Expert Group on 
AI  (AI  HLEG),⁵⁰  a  group  of  52  experts  bringing 
together  representatives  from  academia,  civil 
society, and industry, has been appointed by the 
Commission. Their work includes the elaboration 
of the ethical guidelines for trustworthy AI and 
a  set  of  recommendations  on  future-related 
policy development.

The White Paper on Artificial Intelligence,⁵¹  
published in 2020, aimed to foster a European 
ecosystem  of  excellence  and  trust  in  AI.  This 
initiative  incorporated  a  public  consultation 
that  received  over  1,200  individual  responses 
from  EU  and  non-EU  citizens,  Member 
States  and  relevant  stakeholders,  including 
representatives  from  the  civil  society,  industry 
and academia. Finally, in 2021, the EC published 
a  Communication  on  Fostering  a  European 
approach  to  AI,  combining  one  of  the  world’s 
first attempts at regulating AI and the revision 
of the Coordinated Plan on AI.⁵²

The Proposal for a Regulation (AI Act) laying 
down harmonised rules on artificial intelligence 
contains  new  rules  to  make  sure  that  AI 
systems  used  in  the  EU  are  safe,  transparent, 
ethical,  unbiased  and  under  human  control, 
categorising  them  by  risk.⁵³  The  legal  text 
considers children as a vulnerable population in 
several parts. For instance, according to Article 
9  (Risk  Management  System)  point  8,  “when 
implementing  the  risk  management  system, 
specific consideration shall be given to whether 
the high-risk AI system is likely to be accessed 
by or have an impact on children”. 

In  parallel,  the  review  of  the  Coordinated 
Plan  on  AI  builds  on  the  strong  collaboration 
between  the  Commission  and  Member  States 
established  in  2018,  in  order  to  achieve  EU 
global leadership in trustworthy AI.⁵⁴   No specific 
mention of children is found in the plan, even if 
AI education is mentioned as a key component 
of the EU AI ecosystem, connected to the Digital 
education action plan mentioned as follows.

2.8.3  The  Digital  Education  Action  Plan 
and audio-visual media services directive

Through  actions  related  to  education  and 
training,  the  EU  supports  citizens  in  acquiring 
a basic understanding of digital and emerging 
technologies  which  include  systems  driven  by 
artificial intelligence (AI). 

The Digital Education Action Plan (DEAP) (2021-
2027) is a renewed European Union (EU) policy 
initiative that has various actions towards this 
aim. 

As part of fostering the development of a high-
performing  digital  education  system,  DEAP 
foresees the development of ethical guidelines 

NOTES
⁴⁷ https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52020PC0767&from=EN

⁴⁸ https://digital-strategy.ec.europa.eu/en/policies/data-act

⁴⁹ https://digital-strategy.ec.europa.eu/en/policies/european-approach-artificial-intelligence

⁵⁰ https://digital-strategy.ec.europa.eu/en/policies/expert-group-ai

⁵¹ https://ec.europa.eu/info/publications/white-paper-artificial-intelligence-european-approach-excellence-and-trust_en 

⁵² https://digital-strategy.ec.europa.eu/en/library/communication-fostering-european-approach-artificial-intelligence 

⁵³ https://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1623335154975&uri=CELEX%3A52021PC0206 

⁵⁴ https://digital-strategy.ec.europa.eu/en/library/coordinated-plan-artificial-intelligence-2021-review 

2121

CHAPTER 2

on AI, and data usage in teaching and learning 
to be ready by the beginning of the 2022 school 
year. The overall aim is to help understand the 
potential that AI-based applications could have 
in  education,  and  to  raise  awareness  of  the 
possible risks.⁵⁵  

On  the  other  hand,  to  enhance  citizens’ 
digital  skills  to  cope  better  with  the  digital 
transformation  of  today’s  society  and  the 
economy,  the  DEAP  foresees  action  to  include 
AI  and  data-related  skills⁵⁶  to  the  update  of 
the  European  Digital  Competence  Framework 
(DigComp).⁵⁷    The  aim  is  to  improve  citizens’ 
sensitivity  towards  potential  issues  related 
to  data  protection  and  e-privacy,  rights,  and 
discrimination and bias – including gender bias 
and disabilities, ethnic and racial discrimination.

A  related  action  to  empower  citizens,  and 
especially children, through media literacy skills 
comes  from  the  Audiovisual  Media  Services 
Directive as the availability of harmful content 
(including disinformation) in audiovisual media 
services  and  new  platforms  has  become  a 
challenge.⁵⁸

It  outlines  that  “measures  taken  to  protect 
the  physical,  mental  and  moral  development 
of  minors  and  human  dignity  should  be 
carefully  balanced  with  the  fundamental  right 
to  freedom  of  expression  as  laid  down  in  the 
Charter on Fundamental Rights of the European 
Union”.  One  of  the  concrete  actions  is  that 
video-sharing  platforms  will  need  to  include 
media  literacy  education  for  young  audiences 
which will be implemented in collaboration with 
European  Regulators  Group  for  Audiovisual 
Media Services.

NOTES
⁵⁵ https://ec.europa.eu/education/education-in-the-eu/digital-education-action-plan/action-6_en

⁵⁶ https://ec.europa.eu/education/education-in-the-eu/digital-education-action-plan/action-8_en

⁵⁷ https://ec.europa.eu/jrc/digcomp

⁵⁸ https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32010L0013&from=EN

2222

CHAPTER 3

3. ANALYSIS OF THE IMPACT OF SELECTED 
AI APPLICATIONS ON CHILDREN’S RIGHTS

This section sets out the existing state-of-the-art research and the scientific evidence 

for three AI-based applications that have already been applied in contexts related to 
children:  (i)  recommender  systems,  (ii)  conversational  agents,  and  (iii)  social  robots.  
We summarise the main findings that are relevant to children’s rights. The main criterion for 
the selection of the specific AI technologies, as examples for this report, was their current 
integration  and  use  by  children,  the  current  empirical  evidence  of  their  impact  on  child 
development and their diverse technical features. While conversational agents rely mainly 
on  language-based  communication,  recommender  systems  might  take  text-based  forms, 
and  social  robots  for  their  part  make  use  of  the  physical  components  of  the  system  for 
communication based in multiple modalities. Research in the three cases highlights certain 
directions that are common, such as the development of systems that adapt to children’s 
individual  characteristics  and  needs.  The  research  challenges  that  are  indicated  in  all  the 
cases  focus  on  the  lack  of  effective  methodologies  for  scientific  research  to  address  the 
urgent need for evidence-based policymaking, as well as the need for longitudinal studies to 
examine the long-term effects of the use of those technologies on children’s development. 

Regarding the emerging opportunities, evidence in all the three use-cases indicates that with an 
appropriate approach in the design, development and deployment of those technologies, their 
use could create opportunities for the expansion and the support of children’s development in 
various ways, with a special focus on personalised learning.

However, research indicates that the use of these applications is correlated with risks that 
need to be addressed, such as children’s privacy, possible algorithmic discrimination and lack 
of fairness which is often connected to the lack of transparency and equity in accessibility. 

The  different  characteristics  of  applications  require  a  fine-tuned  approach  indicating  that 
we cannot have one solution that fits all applications within the various contexts; rather we 
need principles and best practices that provide directions for the multiple characteristics of 
AI-based applications for children and their different uses. 

Lastly, the relevant literature indicates that often research and policymaking on AI for children 
are minimally connected, and so there is a need for multistakeholder collaboration, including 
the participation of children in a meaningful way.

Further  research  for  the  development  of  evidence-based  policy  directions  requires  a  bi-
directional understanding of the impact of these applications on children, in terms of the 
scientific evidence relating to AI technical developments and policy structures.

2323

CHAPTER 3

Child-users  as  one  of  the  target  human 
populations  that  are  being  and  will  be 
affected  by  AI  require  special  attention 
which might be different from AI for adults, 
and  separate  research  sub-fields  have 
been developed especially for children. 

First,  children’s  cognitive  and  socio-emotional 
skills manifest rapid growth and they lack fully-
mature abilities which mean that children are a 
category among vulnerable populations. 

Second,  while  AI-based  applications  are 
increasingly  affecting 
children’s  everyday 
lives,  as  in  the  case  of  personalised  learning 
applications,  or  face  recognition  applications 
for  entertainment,  children  are  rarely  invited 
to  be  part  of  the  decision-making  process  for 
the  development  and  deployment  of  those 
applications. 

Third, children need to be prepared in the most 
appropriate  and  effective  way,  not  only  as 
current and future users of AI-enabled systems, 
but also for the possibility of being involved in 
the  design  and  development  of  such  systems. 
This means that children and young people need 
to  be  given  the  opportunities  to  start  develop 
a  mindset  for  human-centred  ethical  designs 
and certain sets of skills that will allow them to 
reflect critically on the design, development and 
use of robotic technology in every setting.

In this chapter, we present scientific findings at 
the  intersection  of  the  rights  of  the  child  with 
three  AI-based  applications: 
recommender 
systems  (RSs),  conversational  agents  (CAs),) 
and social robots for children.

While these systems come with specific common 
characteristics  and  are  overlapping,  there  are 
several characteristics that render them distinct 
from  each  other.  A  recommender  system  is  a 
type of information retrieval system whose goal 
is  to  suggest  items  from  an  ample  collection 
of  activities  that  meet  the  preferences  of  a 
user. These can be text-based, visual-based or 
speech-based systems and might have various 
kinds of interfaces.

Conversational agents rely on language-based 
dialogue systems and might take various forms 
of embodiment, such as screen-based chatbots 
or  voice-enabled  embodied  devices.  Social 
robots are embodied systems that typically use 
multiple  modalities  for  verbal  and  non-verbal 
communication  with  the  user.  In  addition  to 

language, they use modules such as expressive 
non-verbal behaviours, navigation and action to 
interact with the surrounding environment.

For the selection of the case studies that 
are included in this report, we formulated 
the following criteria. 

First,  we  sought  to  include  case-studies  or 
applications  that  are  diverse  in  terms  of  their 
technical characteristics. As shown in Fig.6, an 
application  might  involve  modules  that  are 
based  on  various  techniques,  such  as  natural 
language  processing,  computer  vision  or 
learning from examples. Some of the modules 
involved by the different applications may also 
(or may not) require or record novel data during 
the interaction with children (e.g. reinforcement 
learning), increasing the risk of privacy leakage. 

Second,  the  case-studies  included  had  to 
involve different types of interactions with the 
end-users, such as embodied or non-embodied 
interaction and verbal or non-verbal interaction. 

Lastly,  since  this  report  draws  on  a  science 
for  policy  orientation,  we  decided  to  include 
applications for which there is already scientific 
evidence  about  their  impact  on  children’s 
development.  As  a  result,  three  case-studies 
were 
included,  namely  conversational  AI, 
recommender systems and robotic artefacts.

We acknowledge that with the selection of the 
above-mentioned  case  studies,  we  excluded 
others  that  are  equally  important,  such  as 
applications of learning analytics, AI tools used 
to  detect  CSA  (child  sexual  abuse)  online  (i.e. 
new CSAM and grooming), smart toys and other 
applications, which increasingly affect children’s 
development and well-being.

The  selected  case-studies  are  set  out  in  the 
following  subsections.  For  each  case  study, 
we  first  give  an  overview  of  the  application, 
definitions and a technical description which is 
followed by the opportunities and the emerging 
risks  that  are  connected  to  the  specific 
application,  with  a  connection  to  children’s 
rights.  In  the  end,  we  discuss  emerging  future 
directions. 

Tables  2  and  3  provide  a  summary  of  the 
existing  work  of  the  selected  AI-applications 
regarding the emerging opportunities and risks.

2424

CHAPTER 3

TABLES 2 AND 3
Emerging opportunities and risks in relation to children’s rights, in the context of recommender 
systems, conversational agents and social robots for children, as found in the relevant literature
Source: EC

AI-based applications

Recommender systems

Conversational Agents

Opportunities

Social robots

AI-based applications

Recommender systems

Conversational Agents

Social robots

Accessibility

Engagement for learning
Opportunities
Adaptation

Accessibility
Social interaction

Engagement for learning
Health

Adaptation
Transparency

Social interaction
Inclusivity/Diversification

Health
Acceptance

Transparency
Physical interaction

Inclusivity/Diversification

Acceptance

Physical interaction

Risks

Recommender systems

Conversational Agents

Social robots

AI-based applications

AI-based applications

Recommender systems

Conversational Agents

Social robots

Limited accuracy

Augmentation of 
inequities
Risks

Data disclosure / 
Limited accuracy
Privacy violation

Augmentation of 
Overtrust
inequities

Bias / discrimination
Data disclosure / 
Privacy violation
Over-exposure to 
similar content
Overtrust

Inappropriate content
Bias / discrimination

Reduction of child 
Over-exposure to 
autonomy in relation 
similar content
to the system

Inappropriate content

Reduction of child 
autonomy in relation 
to the system

2525

CHAPTER 3

3.1 Recommender systems

A  recommender  system  (RS)  is  a  type  of 
information  retrieval  system  whose  goal  is  to 
suggest items from a large collection that meet 
the  preferences  of  a  user.  RSs  are  complex 
systems  with  different  components 
that 
contribute  to  their  outcome  and  impact,  such 
as data, algorithms or graphical user interfaces. 
State-of-the-art  recommendation  algorithms 
are hybrid as they combine different approaches, 
such  as  collaborative  filtering  techniques  (i.e. 
recommending to a user the items that a similar 
user liked in the past), content-based methods 
(i.e.    recommending  to  a  user  items  similar  to 
those she/he likes), demographic systems (e.g. 
targeting  specific  languages  or  countries)  or 
knowledge-based  approaches  (e.g.  case-based 
reasoning systems) (Ricci et al., 2011). 

RSs  are  used  in  a  variety  of  domains,  with 
well-known applications such as video services, 
product  recommenders  in  online  shopping, 
content recommenders in social media and web 
content  recommenders  in  different  areas  such 
as  restaurants,  wines,  dating,  news,  language 
teachers or financial services.  

Children  are  common  users  of  recommender 
systems.  Watching  videos  is  one  of  the  most 
common  digital  activities  of  children  reported 
in  the  literature  (Radesky  et  al.,  2020),  where 
tablets  seem  to  be  their  favourite  devices 
according to studies carried out in Europe and 
USA  (Chaudron  et  al.,  2018;  Izci  et  al.,  2019). 
Although  general  RSs  are  widely  used  by 
children, some research studies have addressed 
specific children’s needs, challenges and risks of 
this kind of technology (Fails et al. 2017), and 
companies have also adapted their products to 
children (e.g. YouTube Kids or Spotify Kids). 

In  this  section,  we  briefly  reflect  on  the 
opportunities,  challenges  and  risks  of  RSs  for 
children. We refer to Gómez et al. (2021) for a 
more detailed description of the use case in the 
extensive scientific literature.

Opportunities

The literature identifies several domains where 
recommender  systems  can  bring  value  and 
support children’s autonomy in numerous tasks, 
by  facilitating  access  to  different  information 
sources and modalities. These domains include 
information  search,  media  recommendation, 
story  and  book 
toys, 
learning, 

smart 

recommendations  and  social  media.  RSs  offer 
children  different  opportunities  for  learning, 
play and entertainment, and their functions can 
be summarised as follows: 

▶ Provide access to large sets of material, such 
as in the context of school-based activities or 
entertainment. 

▶ Support the analysis of data to mitigate mis-
information, especially in learning contexts. 
Children  may  have  the  perception  of  being 
able  to  judge  the  credibility  of  information 
sources.  However,  it  may  not  be  the  case, 
even  for  adults.  RS  can  account  for  mecha-
nisms  to  support  children's  interaction  with 
digital  resources  addressing  misinformation 
(Spezzano, 2021).

▶ Support children’s diversification by allow-
ing  each  child  to  control  their  own  learning, 
play  and  entertainment  trajectories,  by  se-
lecting from a large set of recommended ma-
terial.  This  own  control  is  related  to  children 
trusting into RS, which is a relevant topic to 
note, as children may not always rely on RSs 
(Pera et al., 2019).

▶  Facilitate  peer-to-peer  recommendations 

(Picton, 2014). 

▶ RSs used in educational setting can support 
cognitive  self-regulated  learning  skills 
(Tsiakas et al., 2020).

▶  RSs  can  provide  personalised  scaffold-
ing  and  adaptation  for  children’s  learn-
ing  through  recommendations  (Ashlee  et  al., 
2019; Aisha Yaquob et al, 2019).

▶ Monitor and report the child’s progress and 
predict  future  performance  (Ueno  and  Mi-
yazawa, 2017), which might prove beneficial 
for the teaching process. 

Challenges

Ekstrand (2017) summarises the challenges of 
evaluating RSs with children, confirmed by other 
authors,  and  consisting  of  these  three  main 
aspects:

▶ Data availability: the lack of data (i.e. the 
so  called  “cold-start  problem”)  is  one  of  the 
limitations  of  children-centric  studies,  com-
pounded by restrictions due to rights of data 
protection (Milton, 2017). Together, these lim-

2626

CHAPTER 3

it  the  availability  of  benchmarking  datasets 
including for child users, yet which are crucial 
for  algorithm  evaluation  and  development 
and to ensure the reproducibility of studies.

▶ Limited survey abilities when dealing with 
children. Surveys provide a common strategy 
and  practical  way  for  large-scale  evaluation 
of RSs. However, some studies have signalled 
the  limitations  of  this  methodology  for  chil-
dren  (Borders  et  al.,  2000;  Chaudron  et  al., 
2018; Ekstrand, 2017). Other methodologies 
such  as  user  studies,  usability  exercises  and 
participatory  design  processes  are  then  re-
quired for children, which are costly to be car-
ried out on a large scale.

▶ Multi-stakeholder  evaluation: RS evalua-
tion has been traditionally centred on metrics 
and  protocols  that  measure  how  the  differ-
ent system components impact the "user". In 
child-centred  recommendations,  we  need  to 
consider  different  stakeholders,  e.g.  children, 
parents, educators, RS providers. 

Motivated  by  these  limitations,  Gómez  et  al., 
(2021) propose a multi-perspective evaluation 
framework for children-centred RSs, which cover 
four  different  dimensions:  system  component, 
stakeholder,  methodology  and  temporal  scale. 
This  framework  is  designed  as  a  means  for 
the  classification  and  mapping  of  reproducible 
and  incremental  evaluation  practices  allowing 
for  the  scientific  understanding  of  the  impact, 
potential  bias  and  needed  adaptations  of  RSs 
for children.

Risks

While the use of RSs by children brings certain 
opportunities  for  children,  recent  research 
literature,  policy  reports  and  press  articles 
have identified several risks that children may 
encounter  when  using  recommender  systems 
(Chaudron et al., 2018; Izci et al., 2019). These 
risks include:

▶ Personal data processing. One of the risks 
that  recommender  systems  entail  concerns 
the violation of children’s privacy due to per-
sonal data processing. It is important to com-
ply  with  the  regulatory  framework  of  data 
protection,  following  data  protection  princi-
ples (e.g. considering which data are appropri-
ate to collect considering needs), and adopt-
ing  data  protection  by  a  design  approach, 
when designing and developing RSs for chil-

dren. Considering that the target audience is 
children,  particular  care  should  be  added  to 
protecting  the  personal  data  processed  and 
ensuring  that  children  (or  their  legal  tutors 
depending on age) understand and can exer-
cise  their  data  protection  rights  (e.g.  provide 
child-friendly informed consent).

▶ Over-exposure: several studies mention the 
risk of children being exposed to the same or 
similar  content  repeatedly,  because  recom-
mender  systems  are  driven  by  the  notion  of 
similarity. This includes so-called information 
bubbles, understood as the risk for children to 
encounter a certain type of content, based on 
their  previous  choices,  reinforcing  these  and 
so giving children less opportunity and room 
for discovering something different. 

▶  Exposure  to  undesirable  content,  given 
that  entertaining  contents  (e.g.  videos)  are 
not  always  adequate  for  children  and  may 
for  instance  contain  sexual  content,  include 
physical violence or refer to unhealthy food or 
habits. It is important to note that the detec-
tion and filtering of undesirable content is not 
a trivial task. Although some search engines 
have  already  incorporated  such  functional-
ities  (e.g.  “safe-search”),  some  limitations 
have been identified. First, they depend heav-
ily  on  curation,  which  is  not  always  possible 
to  implement  in  a  proper  way.  Second,  they 
may suffer from over-filtering: i.e. preventing 
resources that are suitable but contain (out of 
context) misleading terms. Finally, they most-
ly focus on sexual content, overlooking other 
important  undesirable  content  such  as  hate 
speech,  bullying  or  violence  (Anuyah  et  al., 
2019). Beyond undesirable content, the trade-
off between what children want versus what 
they  need  is  a  challenge  that  must  be  con-
sidered  in  the  design  and  evaluation  of  RSs. 
In  the  educational  context,  for  instance,  this 
trade-off has to be considered together with 
others: e.g. curriculum limitations, commercial 
expectations,  classroom  requirements,  par-
ents’  beliefs  in  the  case  of  educational  con-
text (Murgia et al., 2019).

▶ Online advertising, as platforms may treat 
children as “young consumers”, linked to the 
concept  of  the  “commodification  of  child-
hood”.

▶ Addictions or dependency on screen, derived 
from  the  fact  that  RS  algorithms  maximise 
user engagement with the content. 

2727

CHAPTER 3

▶ Social media platforms or apps such as Mes-
senger  Kids  allow  children  to  post  and  mes-
sage friends through a federation mechanism 
monitored  by  parents.  Some  voices  have 
signalled the risk of these applications to be 
used to familiarise children with commercial 
products that will be used when they become 
teenagers.

▶  Difficulty  for  parents  to  monitor  children’s 
behaviour, as recommender systems are con-
sumed by children mostly on personal devices 
such as tablets or phones. 

▶  Propagation  of  existing  gender  stereo-
types present in search and recommendation 
systems. Recent research works address gen-
der stereotypes in RS (ALRossais and Kidenko, 
2018; Wang et al., 2021), and a position paper 
by Raj et al. (2021) exemplifies some poten-
tial issues related to gender stereotypes con-
cerning children. However, research is needed 
to see if these stereotypes are indeed present 
and propagated in RSs for children, and to de-
fine proper methodologies for their evaluation 
and mitigation.

Although  some  of  these  risks  also  appear  in 
the  adult  population,  children  need  special 
protection, given their vulnerability and potential 
impact  in  their  cognitive  and  socio-emotional 
development.  In  addition,  the  tendency  of 
children  to  use  trial-and-error  methods  to 
learn how to use a tool increases several risks 
such as straying from suitable to non-suitable 
content,  the  accidental  disclosure  of  personal 
information,  and  the  unintended  contact  with 
people.

We  think  that  only  by  evaluating  RSs  from  a 
wide  range  of  perspectives  (i.e.  from  single  to 
longitudinal studies, combining quantitative and 
qualitative methodologies), will we understand 
the  effect  that  their  designs  may  have  on 
individual  stakeholders  (e.g.  children,  parents 
and  businesses),  as  well  as  the  ways  that  in 
which these communities exercise their agency 
and shape these technologies.

3.2 Conversational agents

conversational 

In  the  context  of  this  report,  we  consider  the 
definition  of  a  conversational  agent  (CA)  as 
language-based  computer  program  that 
a 
supports 
interactions  with 
humans. This system is traditionally composed 
of  different  modules  (Fig.2):  automatic  speech 
recognition  (ASR),  transforming  audio  inputs 
into  text;  natural 
language  understanding 
(NLU),  understanding  the  input  text;  dialogue 
manager  (DM),  managing  the  conversational 
agent  actions;  natural  language  generation 
(NLG), translating the computer intent to a text; 
and  text  to  speech  (TTS),  transforming  a  text 
into an audio output. 

Nowadays many systems use Neural Networks 
as  a  particular  module  or  even  for  unifying 
some of them (McTear, 2020).

CAs are accessible and popular among children. 
Studies  about  the  use  of  conversational 
agents, particularly in home environments have 
identified  the  relevance  of  these  devices  for 
children (Sciuto et al., 2018; Garg & Sengupta, 
2020 and Lovato, Piper & Wartella, 2019).

Future directions

Opportunities

This  use  case  summarises  existing  research 
related  to  the  evaluation,  opportunities,  risks, 
and challenges of children using recommender 
systems.  An  analysis  of  the  literature  reveals 
the  importance  of  children-centred  design  to 
minimise the risks that recommender systems 
pose, without sacrificing the opportunities such 
systems can bring to children. 

This  review  also  shows  the  need  to  include 
different perspectives for the evaluation of such 
systems as whether the tools scaffold children’s 
well-being  and  development  by  prioritising 
their  innate  characteristics  such  as  curiosity, 
exploration and creativity.

The  literature  identifies  several  opportunities 
regarding the use of CAs by children:  

▶  Improvement  of  accessibility:  for  children 
too  young  to  write,  with  dyslexia  or  physical 
disabilities  (Pradhan  et  al.,  2018)  (Catania  et 
al., 2021). 

▶  Engagement  of  learning:  information  search 
(Downs  et  al.,  2019)  (Landoni  et  al.,  2020) 
teaching languages (Kanda et al. 2004) (Nasihati 
et al., 2018) or teaching school material (Xu and 
Warschauer, 2020) (Law et al., 2020). 

▶  Promotion  of  social  behaviour:  improving 
persuasion  (Fraser,  Papaioannou  &  Lemon, 

2828

CHAPTER 3

Other  risks  that  have  been  identified  due  to 
misinformation  about  what  a  conversational 
agent is, and how it works, generating over-trust: 
i.e.  children  tend  to  perceive  conversational 
agents  as  friends  when  embodied  in  a  social 
robot (Kahn et al., 2012) or an intelligent speaker 
(Druga et al., 2020). These have high influence 
on  children,  risking  data  disclosure.  In  a  study 
also involving an intelligent speaker, Straten & 
Caroline, et al. (2020) sought to deal with over 
trust through transparency in the programming 
of their systems.  

Gender-specific  cues  are  commonly  used  in 
the design of chatbots and most chatbots are 
–  explicitly  or  implicitly  –  designed  to  convey 
a  specific  gender  with  most  of  them  being 
identified  with  female  names,  female-looking 
avatars, and are described as female chatbots 
(Feine, 2020).

One of the challenges of preventing the risks of 
these devices is that CAs are new and different 
from what we have experienced before. 

2018) (Keizer et al., 2017), and helping autistic 
children (Ali et al., 2020) (Zhang et al., 2020) 

▶ Support of health at home: helping to record 
treatments  and  track  disease  (Sezgin  et  al., 
2020),  as  well  as  reducing  depression  and 
anxiety (Fitzpatrick, Darcy & Vierhile, 2017). 

Risks

Because of all the previous challenges, CAs can 
show  poor  accuracy  when  conversing  with 
some children, depending on their demographics 
(age, socioeconomic background, or language). 
This  fact  affects  their  participation  rights 
and magnifies inequities in our society. 

improving 

child-computer 

That  is  why  a  lot  of  work  has  been  done 
on 
interaction: 
Lavechin  et  al.  (2020)  have  developed  speech 
identification  for  babies;  while  Røyneland 
(2020) and Cheng et al. (2018) have identified 
good strategies to follow when a system does 
not understand a child.  

FIGURE 2 
Block diagram of conversational agents (Adapted from Mansfield et al., 2019)
Source: EC

“What game can we play?”

ASR

NLU

Intent: Game suggestion
Slots: (players:2)

Game: tic-tac-toe

DM

Backend

Inform $Game

TTS

NLG

“What about tic-tac-toe?”

2929

CHAPTER 3

Our inexperience with them makes it difficult to 
prevent future problems. 

An example of an unexpected problem is when 
parents  asked  Amazon  to  change  the  wake-
up  word  “Alexa”  in  their  CA  product,  as  their 
daughters suffered bullying when they had the 
same name (Johns, 2021). 

harm.  We  need  to  pay  special  attention  to 
children,  as  they  are  frequent  users,  and  they 
have special needs. 

The  development  of  trustworthy  CAs  seems 
indispensable to pushing the balance between 
technology and society in favour of users.

Challenges

3.3 Robotic systems

CAs  are  mainly  designed  for  adults,  and  this 
generates some challenges when conversational 
agents  interact  with  children  (Narayanan  & 
Potamianos,  2002)  (Kennedy  et  al.,  2017). 
Different issues have been identified here: 

▶ Children’s speech features differ from adults, 
as they may have a higher pitch range and a 
different prosody. This impacts the ASR mod-
ule (which transforms audio inputs into text).

▶  Children's  expressions  may  also  differ  from 
adults, as they may make grammar or vocab-
ulary mistakes, as well as making up words. 
This  affects  the  NLU  module  performance 
(which understands input text).

▶  Children's  rights  differ  from  adults  as  well, 
as small children should have a different re-
sponse than an adult when they try to access 
a particular service, e.g. buy something online 
or access some specific websites. The right of 
purchase or the right of protection is different. 
This should influence the DM module, which is 
the one driving the conversational agent ac-
tions.

Children’s understanding diverges from adults, 
as they may need simpler words or explanations. 
This  impacts  the  NLG  module  performance 
(which  translates  computer  intent  into  text 
output).

Future directions

We  give  a  short  technical  description  of 
conversational agents and identify the relevance 
of these systems with children. We follow with 
an analysis of opportunities these devices can 
bring  to  kids  and  follow  with  some  challenges 
as CAs may face when talking to a child. Finally, 
we  discuss  some  risks  conversational  agents 
may cause children. 

As with every technology, conversational agents 
can benefit people, but they also cause potential 

A  robotic  system  is  an  autonomous  or  semi-
autonomous  machine,  capable  of  sensing  its 
environment,  carrying  out  computations  to 
make  decisions,  and  performing  actions  in  the 
real world with the use of multiple modalities. 
With  developments 
in  AI,  sensing,  visual 
processing and mobility robots appear in spaces 
that are occupied by humans, including children, 
affecting the ways humans behave and develop. 

Robots  for  children  fall  under  two  general 
paradigms: (i) robots that function as a physical 
interface for children’s programming skills and 
computational  thinking  which  has  a  relatively 
long history in the area of STEAM education (e.g. 
Pappert, 1970); and (ii) robots that are designed 
to interact with children in a socially-meaningful 
way. Although in some cases the line between 
these  two  paradigms  is  blurred,  this  section 
mainly focuses on the second category which is 
known as social robotics. 

For  the  purposes  of  this  report,  we  use  the 
term  “social  robots”  to  refer  to  embodied,  AI-
enabled  systems  with  embedded  sensors  and 
actuators, capable of perceiving and processing 
human social signals, such as human intention 
communication, and complex social structures, 
and responding to them in a socially meaningful 
way (Dauntenhahn et al., 2007). 

Recently, there has been a special interest in the 
impact of the design and deployment of robots 
on child-users. However, the field of child-robot 
interaction is still highly diverse, changing and 
with a scattered set of empirical results. While 
it shares similar gains and concerns with other 
implications  of  AI,  the  embodied  nature  of 
robotic  systems  allows  for  a  set  of  additional 
or different forms of interaction with humans in 
terms of perception and action. 

3030

The  embodiment  of  the  systems  supports  the 
integration of AI into human physical rather than 
virtual environments, which has the potential to 
be embedded in human everyday activities and 
social fabric.

Opportunities

Scientific evidence from empirical studies about 
the  impact  of  robots  on  children’s  behaviour 
demonstrates several examples of applications 
where robots may have an added value. Despite 
several 
technological  and  methodological 
limitations,  the  field  demonstrates:  a  rich 
representation of different age groups; methods 
such as experimental, participatory design and 
ethnographic approaches; and the presence of 
various robotic platforms in formal and informal 
settings. 

the  empirical 

studies 
An  overview  of 
demonstrates  a  consensus 
the 
potential  beneficial  use  of  robots  in  creating 
novel  opportunities  for  children’s  development 
regarding the following areas:

regarding 

▶  Social  learning.  Pedagogical  theories  indi-
cate the importance of children’s social inter-
action  for  learning.  Due  to  the  effectiveness 
in engaging children’s social cognition, robots 
can be used to successfully scaffold children’s 
learning  and  entertainment.  Social  robots 
can  effectively  facilitate  children’s  language 
learning (e.g. Neumann et al., 2020; Johal et 
al., 2020, Vincent-Lancrin et al., 2021), prob-
lem  solving  (Charisi  et  al.,  2020a;  Charisi  et 
al., 2021a).

▶ Children’s perceptions and acceptance of 
robots. Children’s perceptions of their accept-
ance of robotic artefacts have been widely re-
searched, especially in post-intervention ses-
sions  with  various  methodologies.  Research 
shows  children’s  perception  of  robots  to  be 
relatively  positive,  curious  and  exploratory, 
while  there  are  some  differences  in  relation 
to the embodiment of the robot (Søraa et al., 
2021).

▶ Personalisation and adaptation. Children’s 
individual  differences  in  terms  of  cognitive 
and  socio-emotional  development,  as  well 
as their personal needs and preferences can 
be  addressed  with  autonomous  robots  that 
adapt to individual behaviour and appearance 
(Fitter et al., 2018).

CHAPTER 3

▶  Embodied  cognition  and  physical  inter-
action. The embodied nature of robotic arte-
facts facilitates children’s interaction with the 
physical  world,  including  the  physical  social 
interaction  with  robots  and  with  peers.    The 
potential of multimodal, verbal, and non-ver-
bal  communication  of  robots  provides  the 
means  for  a  powerful  interaction,  which  can 
elicit communicational properties that respect 
children’s holistic development. The variety of 
robots’ morphological characteristics and the 
degree of a robot’s anthropomorphic features 
in terms of appearance and behaviour provide 
a large palette of possibilities.

▶ Inclusivity. Extensive research on the use of 
robots  for  diagnostic  or  therapeutic  purpos-
es  with  autistic  children  has  shown  that  the 
robots’  socio-technical  nature  and  their  pre-
dictability are effective for the scaffolding of 
interventions that help autistic children devel-
op social skills (Ghiglino, 2021).

Challenges

While  research  in  the  field  of  child-robot 
interaction  has  already  yielded  evidence  to 
demonstrate the effectiveness and the potential 
benefits  of  the  use  of  robots  in  formal  and 
informal settings with children, there are several 
challenges that require special attention when 
researching  and  deploying  robots  to  interact 
with children. 

▶  From  a  research  perspective,  the  current 
empirical  work 
large  scale  and 
longitudinal  studies  that  span  more  than 
several  months,  such  as  the  one  presented  in 
Davison et al. (2020). 

lacks 

▶  Due  to  current  technical  limitations  in  the 
deployment  of 
robotic 
systems,  most  of  the  empirical  studies  are 
being held in controlled settings with close-
ended tasks.

fully  autonomous 

▶ The vast majority of activities and studies in 
child-robot  interaction  are  being  conducted  in 
developed  countries  with  a  relatively  small 
from  developing  countries, 
representation 
especially  from  Africa  (e.g.  Mondana  et  al., 
2016; Charisi et al., 2021b).

▶  Novelty  effect:  Children’s  behaviour  and 
perceptions of robots are still under the effect 
of  novelty.  That  means  that  children  who 

3131

CHAPTER 3

participate in the research might contribute to 
results that are biased by the novelty.

In addition to the above-mentioned challenges, 
current  evidence  in  research  of  robots  for 
children has highlighted the following concerns:

▶  The  complexity  of  a  robotic  system  which 
relates  to  various  modules  (e.g.  computer 
vision,  decision-making,  planning,  control  etc.) 
which  are  integrated  into  one  system,  makes 
it more difficult to address emerging issues for 
transparency and explainability.

Risks

While 
certain 
robotic  applications  bring 
unique  and  novel  opportunities  for  children’s 
development  and  well-being,  their  design, 
development  and  deployment  entail  certain 
risks.  These  risks  can  be  connected  with 
multiple  aspects,  one  of  which  concerns  the 
characteristics of a robotic application regarding 
its  connectivity,  autonomy,  embodiment  and 
social  embeddedness  (Charisi  et  al.,  2021c). 
We summarise some of the emerging risks as 
follows:

▶ Privacy and data protection. A connected 
embodied  agent  may  collect  and  share  per-
sonal data from and about the child-user. The 
user  might  not  always  be  completely  aware 
of  such  collection  of  processing,  which  rais-
es concerns in terms of transparency (Articles 
12, 23 and 14 of the GDPR) and the informed 
consent that might be required from the data 
subject  (Art.  6  of  the  GDPR).  In  addition,  the 
physical  and  embodied  nature  of  robots  al-
lows  the  combination  of  various  modalities 
that create a more accurate representation of 
the  surrounding  social  and  physical  environ-
ment.  This  might  result  in  the  perception  of 
aspects of human activity that put children’s 
privacy at risk. A social robot might be able to 
navigate the personal space of the child per-
ceiving personal information of child’s private 
environment.  This  might  become  even  more 
complex  in  the  case  of  a  telepresence  robot 
which is manipulated by third users (e.g. so-
cial robots for elderly care).

▶  The  (child’s)  right  to  be  forgotten  and 
other data protection rights. In the case of 
connected robots there are emerging, consid-
erations  regarding  how  the  data  subject  (in 
this  case  the  child)  can  effectively  exercise 
their data protection rights need to be taken 

into  account.  Personal  data  can  be  collected 
and stored in cloud services, where it could be 
kept for longer than necessary and be subject 
to  data  breaches.  In  this  context,  a  personal 
data breach might have a serious impact on 
a child’s future.

▶ Freedom of expression. Children’s interac-
tion with cloud-based robots raises issues of 
their freedom of expression. In the case of a 
child’s interaction with a cloud-based robotic 
device,  it  might  be  advisable  for  a  child  not 
to share sensitive information with a connect-
ed robot. Yet this imposes limitations of chil-
dren’s freedom of expression.

▶ Cyber-security. A connected robotic system 
can be hacked by threating actors, potentially 
impacting the safety and fundamental rights 
of the children interacting with it.

▶  Reduction  of  children  autonomy  and 
oversight.  One  of  the  main  goals  of  social 
autonomous robotic systems is to support hu-
mans in a complementary way and to facili-
tate humans’ well-being. However, in the case 
of over-reliance on robotic support, this might 
raise considerations about human autonomy. 
Especially in the case of child's over-reliance 
on robotic interventions. Bias amplification 
and discrimination system training process-
es  are  often  based  on  the  data  of  humans 
with  a  specific  cognitive,  social  and  cultural 
background. This results in discrimination and 
bias in terms of perception and interaction.

▶ Overtrust. The use of embodied social agents 
has the potential to effectively communicate, 
present  and  deliver  abstract  information  re-
garding the system’s decision-making process 
to  children  in  a  developmentally  appropriate 
manner. Yet, embodied social agents might be 
used as tools for misguidance. A robotic agent 
might trigger children’s trust based on design 
decisions  regarding  its  appearance  and  be-
haviours, which in some cases might lead to 
over-trust.

▶  Propagation  of  existing  gender  stereo-
types.  Robots  may  reinforce  or  counteract 
gender  stereotypes  (Song-Nichols  &  Young, 
2020; Wang et al., 2021) which has an effect 
on human behaviour towards robots (Kraus et 
al.,  2018)  and  eventually  towards  other  hu-
mans. 

3232

CHAPTER 3

▶  Stimulation  of  children’s  agency  attri-
bution  to  machines.  The  human  brain  has 
the  tendency  to  attribute  anthropomorphic 
characteristics  and  agency  to  inanimate  ob-
jects and artefacts. Children attribute agency 
to things that look, move, act and interact like 
agents, by exhibiting goal-directed actions or 
being responsive to changes of their environ-
ments  (Piaget,  1929;  Leslie,  2006).  Agency 
attribution  depends  on  many  variables  and 
how this capability develops from early infan-
cy, and may depend on different factors. For 
example,  agency  attribution  may  be  biased 
by  the  negative  or  positive  outcomes  of  the 
machine (Hamlin et al., 2014), by agency cues 
and by the affective valence of actions’ differ-
ing between cultures (Bart et al., 2019; Goya 
et al. 2019). For this reason, special attention 
should be given to how robots (and they be-
haviours) are designed. 

Future directions

The identification of the potential opportunities 
and of the current challenges and risks for robots 
for child-users is only one first step towards the 
design  and  deployment  of  robots,  which  could 
be used to expand children’s potential in novel 
and safe directions. 

One  of  the  unique  characteristics  of  robotic 
technology is its embodiment and the potential 
embeddedness  in  children’s  everyday  physical 
activities.  For  this  reason,  further  research  is 
needed on how the embodied nature of robots 
might have a distinct impact on children’s rights 
and  how  robots  can  be  used  as  a  tool  for  the 
best interest of all children. 

3333

CHAPTER 3

3434

CHAPTER 4

4. YOUTH’S PERSPECTIVE: FROM THE 
WORKSHOPS WITH CHILDREN AND  
YOUNG PEOPLE

To  study  the  impact  of  AI  technology  on  children’s  rights,  it  seemed  to  us  essential 

to include the voices and views of children in the research process, thus we adopted 
participatory research method. We invited young people aged between 14 and 20 years 
old  from  across  Europe.  Ten  young  participants  coordinated  by  the  European  Schoolnet/
INSAFE network joined. Additionally, as the Spanish ambassador of the Spanish Plataforma 
de Infancia could not be present at the workshop, they provided us with a written contribution 
describing the outputs of two other workshops organised with eight teenagers on the same 
topic.

During the workshop young participants could:

▶ express their views, perceptions and preconceptions of artificial intelligence (AI) and discuss  
   them with experts to explore them deeper; 

▶ determine in what way AI could positively influence children’s rights or, on the contrary, how  
   it could challenge them;

▶ formulate important questions to ask before implementing AI further in our everyday lives,  
   so as to protect and guarantee children’s rights.

Participants recognised the presence and usage of AI technology in usual tools of everyday 
life. They provided several examples in various fields of application such as entertainment, 
gaming,  e-commerce,  education,  transport.  Surprisingly,  apart  from  the  example  already 
provided by Fitbit, they did not report cases of AI applications used in the healthcare system.

Youth  panellists  wished  that  AI  could  contribute  to  get  the  new  generations  to  a  better 
and safer world. This wish took form with some examples that covered different fields of 
application. In some cases, they recognised both opportunities and risks (e.g. online safety 
and security, knowledge and learning). They showed concerns about the amount of personal 
data collected to let AI operate, and acknowledged the tensions in completely trusting AI.

Looking to the future, young participants asked for transparency and explicability (e.g the 
use a child-friendly language) and ex-ante evaluation, involving end-users when conceiving 
and developing AI-based tools. They underlined the need for law and regulation of AI and 
children’s rights to balance protection and participation, and finally they expressed their wish 
to be ready for AI that will surely impact their educational and professional-life paths. 

They called for more knowledge and more (digital) skills for all ages (e.g. children, parents, 
educators)  to  be  accompanied  by  ethical  and  philosophical  questions  when  conceiving, 
developing and using AI.

3535

CHAPTER 4

In this section, we present the youth workshop 
which  provided 
insights  about  children’s 
perspectives on AI and their rights.

To  study  the  impact  of  AI  technology  on 
children’s rights, it seemed essential to include 
the means to collect and study the voices and 
views of children and young people themselves. 
For  this,  we  chose  to  rely  on  the  methods  of 
participatory research (PR) (Vaughn & Jacquez, 
2020).  We  invited  children  and  young  people 
aged  between  14  and  20  years  old  from 
across Europe to discuss with researchers and 
policymakers  AI  and  children’s  rights  in  an 
online workshop. 

Ten  young  participants  joined  JRC  experts 
(AI  technology,  privacy,  online  safety)  in  the 
workshop on the 15 to 16 April 2021: 6 girls and 
4 boys; aged 14 (1), 16 (3), 17 (2), 18 (2) and 
20 (1); coming from Austria (1), Czech Republic 
(1), Greece (1), Iceland (1), Ireland (2), Italy (2), 
Lithuania (2), Malta (1). 

Schoolnet 

coordinated 

their 
European 
participation,  on  the  margins  of  the  INSAFE 
network.  Participants  have  all  been  BIK  Youth 
ambassadors  and  participated  in  the  Safer 
Internet  Forum  in  November  2020.  This  prior 
experience  certainly  raised  awareness  among 
the  young  panellists  of  the  issue  of  children’s 
safety in the digital world.

The  workshop  was  designed  to  allow  young 
participants:  

▶  to  express  their  views,  perceptions  and  pre-
conceptions of artificial intelligence (AI) and to 
discuss them with experts and explore them 
deeper; 

▶ to determine in what way AI could positively 
influence children’s rights or, on the contrary, 
how it could challenge them;

▶ to formulate questions important to ask be-
fore implementing AI further in our everyday 
lives, so as to protect and guarantee children’s 
rights. 

As  a  first  step  in  participatory  research,  this 
methodology  allowed  us  to  collect  children 
and  young  people's  voices  and  views  and  to 
consider them as an important input to shape 
the outcomes of the study.

3636

We  provide  here  an  outcome  summary  of 
the  JRC  youth  workshop,  and  of  two  online 
workshops  gathering  eight  teenagers  in  Spain 
and coordinated by the Spanish Children’s Rights 
Coalition (known as Plataforma de Infancia).

We acknowledge that given the relatively small 
number  of  participants,  the  findings  of  this 
workshop  provide  only  some  indications  for 
further  exploration,  and  we  invite  researchers 
to  expand  on  children’s  perceptions  about  the 
use of AI in various contexts.

4.1 Participants’ preconceptions  
of AI 
Workshop participants recognised the presence 
and usage of AI technology in the usual tools of 
everyday life. They provided several examples in 
various fields of application (Figs.3 and 4). These 
examples can be summarised as follows, based 
on a categorisation by sector of application of 
AI technology, and starting with ones closer to 
the everyday life of the participants: 

Entertainment.  The  use  of  recommendations 
systems based on previous choices powered by 
AI  in  popular  applications  like  Spotify,  TikTok, 
YouTube,  search  engines  and  social  media  in 
general. Items of news or articles written using 
AI  powered  tools.  Machine  drawing  and/or 
enhancing pictures. 

Gaming. BOT and Virtual Non-playing characters 
in video games were mentioned particularly by 
gamers  among  the  participants.    Interestingly, 
speaking  assistants,  such  as  Google  assistant, 
Siri,  Alexa,  were  also  reported  in  this  field  of 
application.  Lego  robot,  even  though  robot 
applications  were  not  discussed  extensively 
as  participants  still  consider  them  as  science-
fiction and non-realistic. 

adjustment 

advertising, 

tools  allowing 
E-Commerce.  AI  powered 
targeted 
of 
prices  according  to  users’  behaviours  and 
interconnections  with  other  services  such  as 
Facebook and Google. A direct experience of AI 
drone grocery delivery services was reported as 
one of those presenting failure. 

Education.  When  thinking  about  education, 
panellists’  thoughts  went  to  the  Covid-19 
pandemic  and  their  experiences  of  remote 
learning  and  online  schooling.  First,  they 
made  some  considerations  on  how  the  entire 

CHAPTER 4

educational  system  has  been  and  will,  most 
probably, be impacted. They see the opportunity 
to  develop  new  ways  of  learning  online  and 
that  AI  could  be  part  of  this.  Then,  they  also 
mentioned that AI systems were already in place 
when using platforms such as Microsoft Teams, 
Zoom  and  others  during  the  lockdown.  Some 
of  the  participants  have  had  some  experience 
with  self-learning  tools  (e.g.  Duolingo),  auto-
correct  keyboards  (i.e.  Gboard),    applications 
that  evaluate  the  users’  level  of  knowledge 
of  a  certain  language  (i.e.  Duolingo)  and 
accommodate the supply of learning materials 
to the level of user’s needs. Linked to this, the 
use of AI within automatic translation services 
from  one  language  to  another  (i.e.  Google 
translator).  To  conclude,  smart  toys  were  also 
named  among  AI-powered  tools  that  could 
support informal learning. 

Transport. Auto driven cars were immediately 
mentioned  by  our  participants.  Along  with  this 
example,  Google  maps  using  predictive  traffic 
was also reported. With a glance to the future, 
participants mentioned the possibility of using 
AI for auto-piloting planes. 

Healthcare. Workshop panellists reported that 
it  would  be  beneficial  if  Artificial  Intelligence 
could  support  future  generations  to  live  in  a 
better world.  AI could serve as a tool to assist 
doctors, either in prevention of diseases and in 
forecasting  future  diagnosis,  or  in  care  during 
surgery.  Fitbit  was  also  among  the  practical 
tools given as an example. Only one of the two 
workshop groups provided an example of AI in 
the field of healthcare. 

Astronomy.  Apart  from  the  use  of  AI  with 
satellites,  no  other  example  was  provided  for 
this specific field of application. 

Agriculture. Panellists identified the use of AI 
for  weather  forecasting,  data  analytics  from 
data collected by drones and for water and crop 
management.

Finance.  Market  analysis  powered  by  AI  to 
make predictions of the evolution of the stock 
market was an example. 

Personal  data  protection.  Participants 
showed concern about their sharing of personal 
data  with  digital  platforms  and  services  in 
exchange  for  free  access  and  use.  They  are 
aware of the trading of personal data that they 

are subject to, and they link this to the need for 
vast quantities of data to develop AI systems.

4.2 Participants’ perceptions on the 
impact of AI on children’s rights
The young panellists provided a large spectrum 
of  examples  and  could  link  them  to  various 
challenges  and  opportunities,  bringing  either 
clearly  positive  outcomes  or  fears  of  negative 
consequences. Most were described challenging 
and  holding  both  positive  and  negative  view, 
depending on the context of an application and 
its  use.  ‘Who’,  ‘when’,  ‘why’,  ‘what  for’  were 
essential dimensions of the discussion.   

In  the  following  sections,  we  present  a 
summary  of  the  benefits/opportunities  and 
risks/challenges  that  AI  can  bring  to  children 
according to our young panellists.  

Benefits and opportunities - “What AI 
can bring to me?”   

As  general  comment,  participants  wished  that 
AI could contribute to bring the new generations 
a better and safer world. This wish took shape 
with  some  examples  that  covered  different 
fields of application: 

Information  and  entertainment.  Bearing 
in  mind  that  recommendations  systems  (RS) 
are  based  on  personal  data  collection,  and 
particularly  the  previous  choices  of  the  users, 
participants  showed  appreciation 
the 
opportunities  that  these  mechanisms  could 
bring to them, notably by easing, personalising 
and reducing search time. In the case of social 
networks  and  media,  RS  support  was  seen 
positively  in  nurturing  social  life  and  opening 
up  opportunities  for  meeting  new  people,  as 
well as getting to know new interesting content, 
tools or products. 

for 

AI  assisted  by  drawing,  picture  or  video- 
enhancing  software  is  part  of  the  common 
toolset  present  on  smartphones  nowadays: 
this  is  fun  and  boosts  creativity.  Personal  and 
speaking assistants like Alexa, Siri or hey Google 
were  also  seen  as  fun,  easy  and  practical  to 
use. Moreover, they offer efficient compensation 
technology  for  non  or  poor  writer-readers, 
including  young  children.  In  the  video-gaming 
world, the use of NPC characters – AI technology 
driven – renders the game more real, enjoyable 
and attractive. 

3737

CHAPTER 4

FIGURE 3
Young participants’ preconceptions and perceptions of AI technology applications (blue team) 
Source: EC

Npc’s

Credit scores 
and manag-
ing risks that 
may occur

Cookies

Data will 
be safed 
in cloud

Filtering 
immedinatly 
harmful content 
without waiting 
for anyone to 
report

Safe 
dating/-
friendship 
apps

Anything 
that can get the 
future genera-
tions to a better 
and safe world

News 
articles 
written by 
AI

Social 
media 
algorithms

Tiktok 
algorithm

i.e. Bot

Data
analysis

GAMING

ASTRONOMY

Market 
analysis 
powered by 
AI-prediction

FINANCE

HEALTHCARE

Fitbit

AI assisted
Diagnosis

Auto-
driven 
cars

Google Maps
using 
predictive 
traﬃc

Weather 
forecasting 
and analytic 
data from 
drones

Water 
and crop 
management

DATA SECURITY

TRANSPORT

AI

ENTERTAINMENT

AGRICULTURE

E-COMMERCE

EDUCATION

Spotify
Recommenda-
tion system 
based on 
previous 
choices

Develop 
the online 
education  
system

Digital 
Advertising

Microso(cid:157) 
teams 
-> distance 
learning -> 
online school

Drone 
delivery 
services

Smart Toys 
(Parenting)

Possible risks as perceived by young panelists

Possible opportunities as perceived by young panelists

3838

CHAPTER 4

FIGURE 4
Young participants’ preconceptions and perceptions of AI technology applications (green team)
Source: EC

AI to screen 
student 
applications 
(CNECT)

Roomba 
vacuum 
cleaners

AI to protect 
websites from 
automatic 
systems 
(ReCaptcha)

Friendship 
suggestions 
in social 
media

AI used to 
screen job 
applicants 

Self-driving 
cars

Google 
recomanda-
tions

Speaking 
assistants 
(e.g. Google, 
Siri, Alexa)

Bot
(in games)

Picture 
classiﬁcation 
(e.g. Google 
pictures, place 
location, face 
recognisiton)

Auto-pilot 
on planes

Lego
robots 

Fit-bit

Weaponized 
Ai (e.g. 
drones)

GAMING

ASTRONOMY

FINANCE

HEALTHCARE

Job 
applications

DATA SECURITY

TRANSPORT

Interconnection 
with other services 
(Facebook's or 
Google's ad 
services)

Targeted 
advertise-
ment

AI

ENTERTAINMENT

AGRICULTURE

Online 
shopping

Prevention 
in health 
care

AI in 
education/-
schools

Spam 
ﬁlters

E-COMMERCE

EDUCATION

AI misuse to 
carry out 
cyberattacks

Adjusting 
prices 
according 
to user 
behaviour

Microso† 
teams 
-> distance 
learning -> 
online school

Satellites

Translations

Online 
platforms 
like Zoom 
and Teams

Self 
learning 
auto-correct 
keyboards 
(Gboard,..)

Duolingo
(evaluates which
grammar topic 
you need 
practice in)

Spotify
Recommendation
system based on 
previous choices

Targeted 
advertisement

Recommendations

Machine 
drawing 
picture - 
picture 
enhacing

Face
detection

YouTube
(ﬁltering harmful 
content may cause 
problems with ad 
revenue which 
people are 
dependent on)

Remote sensing 
(satellite) - for 
concentration 
camp detection 
in China

Ai powered 
surveillance - 
face detection

Possible risks as perceived by young panelists

Possible opportunities as perceived by young panelists

3939

 
CHAPTER 4

Knowledge  and  learning.  AI  could  simplify 
knowledge  discovery  thanks  to  RS  and  voice 
recognition  applied  to  search  engines.  The 
common  use  of  applications  includes:  AI  for 
language  translation  (Google  translator)  as 
practical and easy tools, language learning app 
(Duolingo)  that  allows  more  tailored  learning;, 
AI-based  learning  tools  such  as  editing  and 
auto-corrector  tools  and  self-learning;  and 
predictive  keyboards.  Participants  underlined 
the  promising  opportunities  that  such  tools 
open to students that suffer from difficulties in 
using  traditional  tools  and  learning  strategies 
(dyslexia, visual impaired students, blindness). 

Health and care.  AI could simplify the lives of 
patients  and  doctors  by  enhancing  prevention 
and  assisted  diagnosis.  This  dimension  of 
AI  technology  seemed  still  far  off  from  the 
preoccupations of young participants, although 
they recognised the use of apps and tools such 
as Fitbit as having great potential for preventive 
health  care.  Surprisingly,  apart  from  the  Fitbit 
example already provided, young panellists did 
not report examples of AI applications used in 
the  healthcare  system,  though  the  healthcare 
sector should be especially at the forefront of 
adoption in the EU Strategy for AI. This could be 
explained by the age of participants, but also by 
the way AI is presented to the public: “Maybe our 
generation  is  more  into  fantasy  about  AI  and 
the negative consequences, so the opportunity 
does not come up immediately.”

Online  safety  and  security.  Participants 
realised  the  importance  of  AI  in  cybersecurity, 
and  spontaneously  provided  examples  (e.g. 
protection spam filters, AI supporting the launch 
of cyberattacks). The dual nature of technology 
was  underlined  especially  in  this  part  of  the 
discussion. 

Based  on  the  same  model  that  allows  RS 
tailoring  content  to  be  closer  to  the  user 
thanks to profiling, AI could support prevention 
by  automatically  filtering  harmful  content 
upstream,  rather  than 
leaving  the  users 
reporting  negative  content.  For  example,  in 
areas close to their concerns, participants also 
saw  the  advantage  of  AI  in  providing  secure 
dating/friendship  apps  and  online  educational 
systems. 

Safety  and  security  were  also  discussed  for 
transport  by  self-driven  cars  and  planes,  and 
when talking about finances (credit scores and 
risk management).

Risks  and  Challenges  -  “What  should 
we pay attention to with AI?”

The  discussion  dedicated  to  this  question 
turned  around  AI  usage  and  potential  risks 
to  privacy,  data  protection  and  fundamental 
rights.  Practices  allowing  profiling  were 
perceived sometimes to be close to surveillance. 
Participants  also  mentioned  the  risk  of  failure 
of the AI technology or errors in its outcome, the 
difficulty detecting these errors, and the issue of 
accountability when such errors occur. 

Personal  data  collection.  The  young 
participants  showed  concerns  about 
the 
personal data they exchange for free access to 
platforms and online services. They are aware 
that  the  development  of  AI  systems  needs 
vast  quantities  of  personal  data.  Participants 
aired  their  concerns  as  well  regarding  lack  of 
awareness in: the sharing of personal data (e.g. 
automatic  tagging  and  geo-localisaion);  the 
negative  impact  of  AI  applications  in  terms  of 
data  protection  (e.g.  targeted  advertisement, 
screening  of  job  applicants);  and  about  the 
mandatory  collection  of  personal  data  (e.g. 
cookie walls), as well as complicated and long 
privacy statements not understood by children. 

Trust.  The  trust  that  users  can  invest  in 
AI  technology  remained  at  the 
level  of 
philosophical  questioning  for  most,  having 
them  with  concrete 
difficulties  anchoring 
examples. Still, and close enough to this subject, 
one panellist recognised business opportunities, 
but questioned the trust and the fairness of the 
business  model:  “Also,  the  financial  side.  Lots 
of  people  that  make  money  from  YouTube.  If 
something  is  changed  in  YouTube  through  AI, 
that may affect the finance of those people”.

Surveillance  and  profiling.  They  also 
expressed  concerns  about  weaponisation  of 
AI  in  military  applications  and  surveillance, 
abuse  of  AI  for  marketing  purposes,  but  also 
the  possibility  of  AI  being  used  by  the  State 
to  induce  behaviour  in  people  and  reduce 
their  freedom,  by  “civilising”  them.  Possible 
“positive” applications of AI include avoiding the 
unintended  disclosure  of  personal  data  from 
kids (e.g. in massively multiplayer online games 
(MMOs); filtering email addresses). 

AI  Failures.  Unexpected  AI  failures  (e.g.  AI 
powered content filter that excludes the wrong 
content) and their consequences. 

4040

CHAPTER 4

4.3  What  do  young  people  tell  us, 
what are their concerns?   

Participants  showed  a  balanced  approach 
towards  Artificial  Intelligence.  They  see  AI  as 
an  opportunity  with  potential  benefits  and 
at  the  same  time,  they  expressed  concerns 
about  challenges,  dangers,  risks,  and  harms 
that the use of AI technology can lead to. The 
young  participants  recognised  the  increasing 
complexity of the digital world as an interlinked 
world based on data and information, but also 
as technically challenging. 

AI  –  An  “unavoidable  and  ‘mind  blowing’ 
technology”: Lots of talk, little knowledge. 
Panellists  judged  AI  technology  as  “mind-
blowing” technology, considering its capacities. 
Still their primary concern was that they perceive 
AI as progressing a lot, while they consider that 
people lack knowledge of what AI is and what 
its  implications  are.  Participants  themselves 
declared  knowing  little  about  AI:  “I  don’t  know 
about  AI  intelligence  a  lot  and  I  would  like  to 
learn and maybe talk about AI with my friends 
and  classmates  at  school.  I  want  to  know  like 
the whole situation”. They also recognised that 
AI  technology  is  set  to  become  an  essential 
feature of their lives, yet they questioned how 
fast its adoption will be.

Concerns  on  privacy,  data  protection  and 
trust.  The  discussion  on  AI  technology  and 
data collection generated initial considerations 
on  the  rights  to  privacy  and  data  protection. 
Participants were aware of the trading of their 
personal  data  that  they  are  subject  to  when 
using  online  services,  and  they  link  it  to  the 
need  for  vast  quantities  of  data  to  develop  AI 
systems. 

They  showed  concern  about  their  sharing 
personal  data  with  digital  platforms  and 
services  in  exchange  for  free  access  and 
use.  Furthermore,  they  mentioned  the  risk  of 
attacks  on  people’s  privacy  by  companies  or 
governments (e.g. the use of AI to control where 
any  given  citizen  goes,  and  who  they  relate 
with; the perverse use of facial recognition, GPS 
or data shared by users while on the Internet, 
especially on social media).

The participants linked these privacy and data 
protection  risks  to  the  difficulty  for  young 
people 
information  on  data 
collection  and  data  storage  in  the  legal  terms 

in  accessing 

and  conditions  for  the  common  online  service 
they access (privacy policies), due to a lack of 
use of child-friendly language. One participant 
also  underlined  the  coercive  nature  of  online 
services  (“Who  cannot  be  on  Instagram  or 
TicToc nowadays, if they want to connect with 
friends?”),  and  the  inherent  data  collections 
system at that underpin such services (“Can you 
really say no to data collection if you want to 
use the app?”).

Additionally, the young participants questioned 
the  true  capacities  of  AI,  and  the  impacts 
technology  can  have  on  their  lives.  Regarding 
safety, they questioned the risk of “AI rebelling 
against  humans  and  weaponised  AI”,  as  well 
as  the  risk  of  manipulating  people’s  thinking/
opinions, of children’s behaviours. 

Getting ready for AI – education and the job 
market. Participants showed concerns regarding 
the future of their work and pictured a scenario 
where AI could replace humans. In this context, 
they questioned how the school and society can 
prepare  them  for  a  world  transformed  by  the 
AI revolution in motion: “What to learn, how to 
learn?”. On the technical and skills side, but also 
on the ethical and emotional side, it was noted 
that: “It is not because a technology allows to 
do things that it is good to use it”.

4.4 What do young people propose?     

To  balance  protection  and  participation  in  a 
digital  world  driven  by  AI-technology,  young 
panellists proposed:

▶ More knowledge, more skills - protective rules 
and ex-ante risk assessment. Awareness rais-
ing  and  empowerment  through  education 
among  users  of  all  ages  using  age  adapted 
materials  and  groups  (e.g.  disabled  and  vul-
nerable), on: (i) the nature of AI; (ii) its ways 
of functioning, what it needs, what it provides; 
(iii) if something goes wrong who can help to 
solve it; (iv) on improvements process; and fi-
nally  (v)  on  the  legislative  framework  within 
which AI systems can be used. Young panel-
lists requested children-friendly risks assess-
ment of the technology from designers before 
the market release of applications, based on 
their ethical questioning of AI.

▶  Transparency  and  explicability  -  Finally,  the 
young panellists demanded transparency re-
garding AI: how and where it is used, for what 

4141

CHAPTER 4

aims,  and  based  on  what.  They  also  asked 
for  a  simple  and  child-friendly  explanation 
of the use of AI technology as a way of rais-
ing  knowledge  and  awareness.  They  drew  a 
parallel  with  the  imposition  of  cookie-tech-
nology alerts, which according to them raise 
awareness  about  data  collection  and  shar-
ing.  Drawing  on  this  model,  the  participants 
suggested the creation of AI alert messages 
and  clear  and  simple  information  regarding 
its use, when the technology is used in online 
services.

4.5  Participants’  questions  about 
future directions
We  report  in  Fig.5  the  questions  that  the 
participants  of  the  youth  workshop  found 
important to ask, at this stage of their analysis 
and  understanding  of  AI 
its 
applications,  its  benefits/opportunities,  risks, 
and challenges. 

technology, 

4242

CHAPTER 4

FIGURE 5
Young workshop participants’ questions on future directions
Source: EC

Safety and risks

Future prospect 

Educational needs 
to prepare the 
new generations 
to AI

Employement 

How exactly will it affect people, 
especially their jobs?

What will be needed by our 
generation to be able to know 
exactly how it will affect us and 
how students are going to be 
educated about AI?

Regarding education, will AI be 
able to instruct young children 
about empathy and other 
important emotions?

Would AI be trained very well to 
not turn on their owner?

Since AI can have the capability 
of learning, how can we be sure 
that it is respecting children’s 
rights?

Will AI be able to make the 
situation better or worse?
Will AI be able to manipulate 
people’s thinking/opinions? 

(Assuming we want it to do so 
as little as possible) - How can 
we prevent it from doing so?

Where should the line be 
between marketing and 
children's psychological 
well-being? 

Is AI manipulating children?
What dangers does 
unmonitored AI bring to humans 
using it?

All users need to know more 
about AI – information that they 
can understand should be 
provided.

What may be the timespan till 
AI becomes an essential part of 
everyday life?

Will AI become more smarter, 
more capable etc. in the future?

What is the presumed 
interaction between humans 
and AI is going to look like?

Do people get (grasp) the 
definition of AI?

Do people get (understand) the 
benefits of using AI?

Do people get (understand) the 
risks of using AI?

Are users aware in which parts 
of their life AI is already 
present?

Trust

Future prospect 

Provision: 
Increasing 
awareness and 
feedback to users

Protection and 
Participation

Do people trust AI?

In which tasks do they trust AI 
and in which they do not?

Will AI be able to make the 
situation better or worse?

How reliable can AI really be in 
comparison to humans doing 
the same job?

What is the chance of recently 
developed AI may compromise 
children’s privacy?

Are users aware of the presence 
of AI technology on the online 
platform they decide to visit

Where is the line between 
protection and participation of 
children in AI?

When we feed the AI with data, 
are the users well informed 
about which data it is and for 
what purpose will it be used?

Are children (potentially all 
users) informed about it in a 
clear and easy-to-understand 
way?

Protection is the “base” which 
we should think about first and 
then build upon it further. 
Making children “comfortable” in 
a way that they trust the AI 
system, this also goes 
hand-in-hand with protection of 
children.

Will further regulations be put 
in place? Which ones?

Will they ensure the advantages 
of the free market but also 
respect people’s privacy?

Where’s the middle ground 
between these two

4343

4444

CHAPTER 5

5.  EXPERTS’ PERSPECTIVE: FROM THE 
WORKSHOPS WITH SCIENTISTS AND 
POLICYMAKERS

This chapter proposes a set of directions for the implementation of future research and 

policy on AI and children’s rights. It summarises the outcome of a series of three online 
workshops  at  the  end  of  2021,  that  gathered  eight  JRC  researchers,  eight  external 
experts  with  varied  backgrounds  and  seven  EU  policymakers  (see  the  acknowledgement 
section). After being introduced to the outcomes of the reviews on policy initiatives and on 
the scientific evidence regarding the three selected AI systems as well as to the outcomes 
of  the  youth  workshops,  the  participants  explored  the  needs  and  gaps  in  research  and 
policy  regarding  AI  and  children’s  rights.  They  identified  and  articulated  requirements  and 
methods that need to be considered, as well as knowledge gaps and directions that need to 
be prioritised for future research and policy.

Five requirements were identified: (i) AI minimisation, valuable purpose and sustainability, 
(ii) transparency, explainability and accountability, (iii) inclusion and non-discrimination, (iv) 
privacy, data protection and safety, and (v) integration and respect of children's agency.

Four methods were highlighted as necessary to develop successfully and frame child-friendly 
AI technology: (i) anticipation, evaluation and monitoring, (ii) multi-stakeholder collaboration, 
(iii) children’s participation, and (iv) balancing conflicting rights.

Pressing knowledge-gaps and three priority directions for research were also underlined: (i) 
children’s cognition, development, and play, (ii) empowering through education, and (iii) age 
verification systems.

Here  again,  we  acknowledge  that  given  the  small  number  of  participants  in  the  experts’ 
workshop, the findings presented provide only some indications which merit further exploration, 
and we invite researchers to further explore AI and children’s rights in greater detail..

4545

CHAPTER 5

We  invited  specialists  from  various  scientific 
fields  and  sectors  to  participate  together  with 
EU policymakers in a series of three workshops, 
which  took  place  in  November  and  December 
2021,  and  to  reflect  on  future  directions  for 
implementation of AI in relation to children.⁵⁹ 

The first workshop focused on the introduction 
of  the  scope  of  the  workshop  series,  the 
presentation of the core elements of the current 
report, as well as on generating ideas about the 
topics of the report.

The  second  workshop  considered  the  written 
feedback by the experts on the first draft of this 
report, and considered a series of policy-related 
questions  that  we  received  from  the  invited 
policy  experts.  Based  on  these  questions,  the 
participants were asked to formulate a project 
which would prioritise certain topics and would 
propose corresponding methodologies.

The 
the 
third  workshop  elaborated  on 
conclusions of the activities, as well as on the 
reflections  regarding  the  methodologies  that 
could  facilitate  research  to  support  decision-
making.

Below we present and elaborate on the topics 
that emerged during the interaction of experts 
and  policymakers  in  the  three  workshops, 
and  the  questions  that  were  proposed  for  an 
integrated policy and research agenda.

5.1 Requirements for AI policy 
supporting children’s rights
Five requirements were identified:  

I.  AI  minimisation,  valuable  purpose  and  sus-

tainability, 

II. transparency, explainability and accountability, 

III. inclusion and non-discrimination, 

IV. privacy, data protection and safety, and 

V. integration and respect of children's agency.

5.1.1 AI Minimisation, valuable purposes 
and sustainability 

An important dimension of children’s lives that 
was  only  briefly  tackled  during  the  workshop 
relates  to  the  quality  of  the  environment  in 
which  children  are  growing.  In  its  Art.  27,  the 
UNCRC  especially  takes  into  consideration  the 
risks of environmental pollution.⁶⁰  AI technology 
shows potential to address some environmental 
challenges in several fields, according to some 
publications  (Vinuesa  et  al  2020;  Liu  et  al. 
2019). 

Smart  energy  grids,  smart  farming,  smart 
transport  are  a  few  examples  of  fields  in 
which AI is already showing promising positive 
environmental  impacts  around  the  world.  Still, 
data centres, critical for storing the substantial 
amounts of data needed to power AI systems, 
demand  huge  amounts  of  energy,  and  have 
already  pointed  out  as  important  contributors 
to increased CO2 emissions.⁶¹ 

Recently, Vinuesa et. al (2020) concluded their 
study  on  the  role  of  AI  in  achieving  the  17 
UN  Sustainable  Development  goals  (Strubell 
et  al.,  2019)  by  calling  for  actions  to  achieve 
sustainable  AI  that  go  beyond  environmental 
considerations, to include systemic and ethical 
evaluations.⁶²  Similarly, a recent European report 
on  the  Impacts  of  the  digital  transformation 
on  the  environment  and  sustainability  (Liu  et 
al.  2019)  already  recommended  measures  to 
minimise  the  footprint  of  digital  products  and 
services.  This    echoes  the  above-mentioned 
need  for  more  transparent  handling  and 
protection of personal data. 

Furthermore, an approach rooted in the concept 
of  ‘data  minimisation’  and  ‘AI  minimisation’ 

NOTES
⁵⁹ We invited 8 experts with backgrounds in engineering, developmental psychology, law, child-computer interaction, and robotics as well 
as policymakers from the relevant Directorates of the European Commission (DG.HOME, DG.JUST and DG.CNECT), as well as other policy 
organisations and agencies, such as the European Union Agency for Fundamental Rights. The experts came from diverse geographical 
areas: Belgium, Chile, Italy, Netherlands, Spain, the UK and the USA. 

⁶⁰ https://www.unicef.org/child-rights-convention/convention-text 

⁶¹ https://ec.europa.eu/environment/enveco/resource_efficiency/pdf/studies/issue_paper_digital_transformation_20191220_final.pdf 

⁶² THE 17 GOALS | Sustainable Development (un.org) https://sdgs.un.org/goals   

4646

would limit the use of AI to complete tasks that 
are  considered  essential,  and  having  valuable 
purposes. Yet, this opens up the crucial question 
‘valuable  purposes’  are.  Despite 
of  what 
relevant research in the field, more evidence is 
needed to determine what ‘valuable purposes’ 
are for children. 

in 

Experts  participating 
this  work  stress 
that  society  must  not  consider  the  use  of  AI 
technology as an unlimited resource. Strategic 
and systemic choices will be needed to develop 
the  AI  services  at  public  and  private  levels. 
Considering the negative environmental impact 
and  the  positive  implications  of  AI  on  climate 
change  will  be  crucial  in  moving  forward  to 
better children’s lives.  

5.1.2  Transparency,  explainability  and 
accountability

The  literature  review  and  the  interdisciplinary 
discussions  with  researchers,  policymakers 
and  teenagers  confirm  the  importance  of 
transparency and information provision as a key 
consideration for AI and children. Transparency 
and explainability are related to the right to the 
information  but  also  linked  to  education  and 
literacy. 

Transparency and explainability

that  children  can  become 

As  a  conclusion  of  our  analysis,  transparency 
so 
informed, 
empowered citizens and users of AI is seen as 
a means to promote critical thinking and fight 
misinformation  and  preventing  over-trust,  or 
mistrust in AI systems. 

What  are  the  most  suitable  ways  to  explain 
AI  systems  in  child-friendly  language  remains 
an  area  to  be  researched,  while  being  a  topic 
that  is  partially  being  addressed  now  by 
various  non-profit  educational  organisations. 
This also includes an analysis of how to adapt 
these  explanations  to  children  at  different 
developmental stages.

Aware of the increasing presence of AI in their 
daily  lives,  teenagers  taking  part  in  our  youth 
workshop  mentioned  their  wish  to  learn  more 
about  AI,  and  the  need  to  create  awareness 
and  knowledge  about  the  nature  of  AI,  how 
it  works,  what  it  needs,  what  it  offers,  the 
legal  requirements  for  AI,  how  to  behave  in  a 
responsible way with AI. 

CHAPTER 5

Performance  and  accuracy  of  models  are  no 
longer the only criteria for evaluating algorithms. 
Their 
transparency  and  explainability  has 
become  an  additional  criterion  and  a  priority 
that cannot be ignored, especially for end-users 
(Weitz et al., 2021).

Accountability  

It  is  widely  accepted  that  all  AI  actors  should 
be accountable for the proper functioning of AI 
systems, based on their roles, as well as on the 
context in which they are applied, in prioritising 
children’s  fundamental  rights.  The  involved 
organisations  and  individuals  are  expected  to 
ensure the proper functioning of the AI systems, 
throughout  their  lifecycle,  that  they  design, 
develop,  operate  or  deploy,  in  accordance 
with  their  roles  and  the  applicable  regulatory 
frameworks. 

through 

is  expected 

to  be 
Their  accountability 
demonstrated 
their  actions  and 
decision-making process and it involves: (i) the 
data used to develop and operate an AI model; 
(ii)  the  governance  at  an  organisational  and 
system level; (iii) monitoring of the system over 
time; and (iv) the performance of the system or 
of the components of the system. 

This is highly connected with transparency at an 
organisational and system level of procedures 
that can be audited. However, AI systems pose 
unique  challenges  because  their  input  and 
operations are not always visible, especially to 
the  end-users.  An  AI  system  might  be  opaque 
implemented  algorithms  are 
because  the 
inherently difficult to understand or because of 
proprietary  reasons.  As  such,  especially  for  AI 
systems that are designed for children, there is 
an  emerging  need  for  the  identification  of  the 
minimum requirements that would ensure and 
monitor  the  accountability  of  the  involved  AI 
actors.

5.1.3 Inclusion and non-discrimination
The  challenges  and  risks  of  exclusion  and 
discrimination  due  to  the  use  of  ‘biased  data’ 
have  been  the  subject  of  extensive  debates 
worldwide, and the case of children’s data has 
raised  special  attention  in  this  context.  The 
discussions with experts and youth, preparatory 
to  this  work,  suggest  that  a  unique  approach 
will  not  be  sufficient  to  tackle  the  theme 
comprehensively.  Instead,  a  socio-ecological 
approach seems promising to enhance children’s 

4747

CHAPTER 5

rights  to  inclusion  and  non-discrimination  by 
developing  capacity-building  and  knowledge 
sharing.  Based  on  Bronfenbrenner’s  Ecological 
Systems Theory (1979), we identified four levels 
of 
interactions  (individual/child,  community, 
societal  and  policy  levels)  emphasising  the 
importance  of  the  interplay  and  relationships 
between them (Fig.6).

Child

Drawing  on  the  first  interactions  during  the 
Youth  Workshop,  participants  pointed  out  the 
need for more knowledge, and rules to balance 
protection  and  participation  in  a  digital  world 
driven  by  AI-technology.  They  underlined  the 
concept  of  diversity  and  the  need  to  tackle 
users of all ages (very young children, children, 
adolescents,  but  also  adults)  and  particular 
groups  (e.g.  disability,  special  needs,  cultural, 
socio-economic and gender).

Interestingly, they already suggested the need 
for  interactions  with  the  other  levels  of  the 
model.  More  precisely,  they  highlighted  the 
need  to  build  AI  knowledge  for  their  adults  of 
reference  such  educators  and  care  givers.  As 
the literature suggests (ZOPED-Vygotsky) these 

“more knowledgeable others”, like parents and 
teachers,  may  enhance  the  children’s  learning 
slightly  above  a  child’s  ability  level  or  provide 
support  to  avoid  exclusion  and  discrimination 
“in case something goes wrong” (cit. Panellist).  
The  role  of  education  will  be  dealt  with  more 
extensively  in  the  next  paragraph:  however 
experts  suggested  emphasising  the  relevance 
of formal education to build paths to minimise 
inequalities and foster inclusion. 

Indeed, not only do others matter to empower 
children  about  AI  technology  and  their  rights: 
individuals,  including  children,  should  develop 
their  human  agency  by  acquiring  new  skills 
and  knowledge,  but  also  by  being  curious 
and  questioning  AI  technology  and  how  these 
systems  work.  This  goes  beyond  merely 
teaching,  and  requires  the  internalisation  of 
certain concepts.

Community

The significance of the role that carers can play 
within  the  inclusion  and  non-discrimination  of 
children in their interaction with AI-based tools 
was  already  anticipated  in  the  first  section. 
However,  it  is  important  to  note  that  parents 

FIGURE 6
Brofenbrenner’s Ecological System
Source: EC

POLICY
Rights, Algorithms,
Security

SOCIETY
Educational enities,
Industry, Researchers

COMMUNITY
Parents, Teachers, Peers

CHILD
Knowledge, 
Agency,
Understanding,
Empowerment

4848

CHAPTER 5

are often addressed either to blame them (e.g. 
for being responsible of risks and consequently 
possible  harm)  or  they  are  patronised  (e.g. 
considered  as  the  ones  who  allow  the  use  of 
a  digital  technology  or  not).  Together  with 
parents, teachers are frequently asked to take 
over the burden to teach children in recognising 
risks while using digital technology. 

Relying on parents and teachers does not imply 
any  reduction  in  other  actors’  responsibilities. 
Developers, industry, data controllers, regulatory 
bodies,  policymakers  and  finally  researchers 
play also a crucial role.

Society

The  need  for  further  research  was  underlined 
several times among experts. There is a lack of 
interdisciplinary  research,  longitudinal  studies 
and non-experimental approaches. Suggestions 
for  participatory 
research  where  children 
encounter these technologies was made. It is by 
observing  their  direct  interactions,  the  setting 
(at home or in simulated environment), that it is 
possible to understand how digital inequalities 
come into play and how children’s experiences 
are different based on their age, socio-economic 
context,  digital  skills,  and  kind  of  access  they 
have to technologies. 

The  cautious  involvement  of  industry  within 
research could also be considered in obtaining 
a full picture of the data processing. There are 
also  limitations  related  to  the  unavailability 
of  developers  to  share  data.  Linked  to  this, 
the  use  of  Big  Data  and  the  personal  data  of 
children for the purposes of Machine Learning, 
Communication with Machines and Automation 
deserve particular attention.

There is a need to understand the impact of: i) 
more automation, ii) more data, and iii) machine 
communication. 

Policy

The right to participation, inclusion and non-dis-
crimination was addressed by all policymakers 
attending the workshop in different fields of ap-
plication; starting from education, personal data 
and privacy issues, cognition and development, 
and children’s abuse and exploitation. 

The  common  understanding  is  that  there  is 
a  need  for  a  community-driven  exercise  to 
design those policies that could guarantee the 

respect  of  children’s  rights  in  AI  systems.  This 
could be achieved by simulating intervention of 
the  foreseen  solutions  and  addressing  ethical 
challenges. 

The  recommendation  is  to  develop  concrete 
guidelines  and  an  evaluation  framework  to 
assess the impact of AI technology on children’s 
rights.  As  previously  pointed  out  by  the  youth 
panel,  considering 
risks 
assessment  of  technology  from  designers  is 
also a key challenge.

the  child-friendly 

Algorithmic Bias

Research  and  development  of  AI  systems  for 
children  are  often  carried  out  in  the  context 
of  WEIRD  (Western,  Educated,  Industrialised, 
Rich and Democratic) societies, and we need to 
make  sure  that  technologies,  methodologies, 
conclusions  and  recommendations  can  be 
extended  to  other  populations.  Diversity  in 
terms  of  gender,  age,  socio-economic  status 
and cultural context should be our target goal 
in the development of AI systems. 

Another  important  aspect  is  the  consideration 
of  children  with  disabilities,  as  AI  tools  are 
already  being  used  to  help  persons  with 
disabilities. AI systems have a strong potential 
to support people with vision, hearing, mobility 
and  learning  disabilities.  The  use  of  AI  by 
persons  with  disabilities  can  lead  to  living  a 
more  independent  life.  However,  certain  AI 
outputs  can  reflect  discriminatory  biases  due 
to the algorithms they rely upon, or due to the 
underlying  data  used  to  develop  algorithms. 
One  of  the  main  questions  regarding  children 
with disabilities is how to ensure that the use of 
AI  systems  do  not  systematically  discriminate 
against children with disabilities?

There is a need to ensure that AI systems do not 
systematically reflect discriminatory biases due 
to  the  algorithms  they  rely  on.  It  is  of  utmost 
importance  to  develop  concrete  guidelines 
and  an  evaluation  framework  to  assess  the 
impact  of  AI  technology  on  every  child’s  right, 
considering the child-friendly risks assessment 
of technology from designers and developers. 

 5.1.4 Privacy, data protection and safety
AI  breakthroughs  observed  in  the  last  decade 
are  recognised  to  be  largely  supported  by 
advances  in  computational  capabilities,  a  new 
generation  of  machine  learning  algorithms, 

4949

CHAPTER 5

and  the  processing  of  huge  quantities  of 
data.  Nowadays,  personal  data  is  much  used 
to  both  train  AI  systems  and  as  part  of  their 
in  making  decisions,  predictions  or 
input 
recommendations, or for generating content. 

its 

In this context, AI is closely entangled with data 
protection, as it heavily involves the processing 
of  personal  data  during 
lifecycle.  AI 
technologies  enable  new  levels  of  automation 
and new functionalities, allowing for new types 
of  practices,  many  of  which  take  place  in  the 
context  of  interactions  with  children.  This  has 
become  the  core  business  of  many  digital 
companies and platforms.

The  young  participants  in  our  preparatory 
work  linked  privacy  and  data  protection  risks 
to  the  difficulty  for  young  people  of  accessing 
information  on  data  collection  and  data 
storage,  given  the  complex  language  in  which 
legal  terms  and  conditions  of  the  common 
online  services  they  access  (privacy  policies) 
are provided. Young participants also underlined 
the coercive nature of online services and of the 
inherent data collection systems on which they 
are based, questioning whether one can really 
avoid  data  collection  in  using  certain  online 
applications.

The  specific  nature  of  AI,  considering  its 
opaqueness  and  complexity,  poses  additional 
risks  on  top  of  those  that  already  relate  to 
privacy  and  data  protection,  connected  to 
the  principles  of  fairness  and  transparency, 
accuracy, purpose limitation, data minimisation 
and the storage limitation of the GDPR.⁶³ 

The GDPR is an important pillar to support and 
manage  this  new  AI  powered  revolution,  in 
preventing  negative  impacts  on  fundamental 
rights.  In  this  context,  cybersecurity  is  also 
equally important to ensure that the uptake of 
AI-enabled products and services do not result 
in further risks to children, including safety.

The  GDPR  recognises  that  children  deserve 
special  protection  when 
it  comes  to  the 
processing of their personal data. In the context 
of  AI  technologies,  this  has  become  especially 
relevant.  Children  need  to  be  empowered  to 
control how their personal data is used and what 

the implications are. There has to be an effort, 
as it is mandated by the GDPR, to make privacy 
statements  understandable  by  children,  and 
support them in exercising their data protection 
rights. More generally, data protection principles, 
such as privacy-by-design, should take children 
into  consideration  and  translate  into  effective 
mechanisms to ensure their protection. 

Integration and respect of children’s right 
for agency 

Children’s  right  of  agency  is  one  of  the 
fundamental  requirements  to  be  met  when 
designing and using AI for children. Respecting 
children’s  right  for  agency  means  that  the 
option for the no-use of AI should be given to 
the children, while at the same time we ensure 
that the children choosing off-line and non-AI-
based  activities  have  equal  opportunities  for 
development. 

Adding  to  this  challenge,  research  usually 
considers  children’s  perceived  agency  as  a 
proxy for their actual agency. However, agency 
is a complex phenomenon which consists of at 
least a three-levels process: 

▶ the child’s sense of agency, 

▶ the opportunities provided to a child for the 

exercise of agency; and 

▶ the transformation of opportunities into affor-
dances,  by  designing  systems  that  facilitate 
children  to  recognise  consciously  the  oppor-
tunities – it is affordances, not just opportu-
nities,  that  are  crucial  prerequisites  for  the 
exercise of agency by children. 

During the workshops, the experts indicated the 
need for further research on how the construct 
of  agency  is  developed  in  children,  when  they 
interact with AI-based systems.

NOTES
⁶³ https://www.europarl.europa.eu/RegData/etudes/STUD/2020/641530/EPRS_STU(2020)641530_EN.pdf

5050

CHAPTER 5

5.2. Methods to develop AI policy 
supporting children’s rights

Four methods were highlighted as necessary to 
develop  successfully  and  frame  child-friendly 
AI technology: 

I. Anticipation, evaluation and monitoring, 

II. Multi-stakeholder collaboration, 

III. Children’s participation, and 

IV. Balancing conflicting rights.

5.2.1 Anticipation, evaluation and 
monitoring

Research  dedicated  to  the  evaluation  and 
monitoring AI technology used by/designed for 
children are most of the time limited in terms 
of  case-studies,  stakeholders  and  the  period 
considered. The importance of multi-stakeholder 
and multi-perspective evaluation methods was 
mentioned in the literature and by the experts 
consulted.  Children-centric  evaluation  should 
including 
combine  different  methodologies, 
participatory methods, interdisciplinary research, 
collaborative  consultation  of  stakeholders  and 
large-scale studies. 

It should also consider different temporal scales 
and  stakeholders.  Evaluation  also  needs  real-
world scenarios, where the role of industry was 
considered as involving key participatory actors, 
in  collaboration  with  academia,  civil  society 
actors,  policymakers  and  children.  Identified 
research  questions  include  the  creation  of 
frameworks and toolkits that can enable/guide 
the design and evaluation in the short and long-
term of AI systems having children as their users. 
These  frameworks  should  incorporate  aspects 
such as data protection and risk assessment.

5.2.2 Multistakeholder collaboration
During the workshop, it was highlighted that the 
collaboration among stakeholders from different 
sectors, and the inclusion of children as actors 
are important for the democratic development 
of  AI-based  technology  for  children.  However, 
presents 
multistakeholder 
major  challenges.  For 
instance,  different 
stakeholders  might  have  different,  sometimes 
conflicting,  goals,  aims  and  strategies.    Also, 

collaboration 

they might consider key concepts differently in 
creating  interferences  in  communication  and 
interactions.  The  experts  consulted  proposed 
that  the  future  AI  research  agenda  should 
include  the  outline  of  a  simple  framework  (or 
guiding  questions)  and  that  would  facilitate 
communication  among  stakeholders.  Lastly,  it 
was suggested platforms should be developed 
that would facilitate the interaction of different 
stakeholders  based  on  common  projects,  but 
also  a  common  conceptualisation  (a  lingua 
franca). Children should be actively involved in 
such work.

5.2.3 Children’s participation
Some  of  the  above-mentioned  topics  for 
future  research  and  policy  have  already  been 
extensively  researched  with  the  use  of  well-
established  methodologies  from  neighbouring 
fields.  For  example,  the  use  of  experimental 
studies  is  a  widely-used  methodology  for  the 
examination  of  the  impact  of  certain  design 
decisions  on  children’s  cognitive  processes. 
However, there are other areas that are novel, 
or  situations  that  are  unique  because  of  the 
nature  of  AI-systems  and  that  require  novel 
methodological approaches.

rights 

For  example,  children’s 
for  non-
discrimination require an AI system to perceive 
and  meaningfully  process  information  from 
children  with  different  cognitive  stages  or 
socio-emotional  behaviours.  In  such  a  case, 
an  AI  system  should  not  discriminate  against 
minorities. One of the methodological decisions 
that would mitigate discrimination and promote 
fairness  among  children  is  the  inclusion  of 
under-represented populations. 

There  are  already  initiatives  that  propose 
certain  methodologies  for  the  investigation 
of  the  impact  of  AI  on  human  behaviour  such 
as  the  “IEEE:  IEEE  Recommended  Practice 
for  Assessing  the  Impact  of  Autonomous  and 
Intelligent Systems on Human Well-Being. IEEE 
Std 7010-2020, pp.1–96”. 

However,  special  attention  should  be  given 
when  researching  with  children.  For  example, 
because  of  the  children’s  fundamental  and 
unique characteristic of rapid development, it is 
important for research methods to focus on the 
developmental process over time. 

Longitudinal  studies  are  needed  to  provide 
evidence on the impact of AI on children over the 

5151

CHAPTER 5

Children’s participation with a focus on inclusion and diversity.

In 2020, the JRC piloted UNICEF's Policy Guidance on AI for Children in 

collaboration with the Honda Research Institute, Japan. For this pilot, we 
invited children with a diverse cultural background focusing on the 
inclusion of children from typically under-represented geographical 
areas, namely Uganda, Japan and Greece. The proposed requirements were 
implemented into a robotic prototype platform, called Haru, which is being 

developed by the Honda Research Institute. This is a table-top social robot 

that can be used to support children’s cognitive and social development with 

a design based on animation principles.

We used Participatory Action Research to provide the space for the local 
community to talk about their values, interests and needs. The educators 
and the children became part of the research team and interacted with the 
researchers and the developers. This helped us gain a deep understanding 

of the ecosystem in which AI system will be integrated in the future and, at 

the same time, the local communities were an integral part of the global 

dialogue about AI for Children (Charisi et al. 2021b; UNICEF, 2021).

long term. This is often a subject of contention in 
the online safety/well-being field, where there is 
a tendency to prioritise immediate impressions 
of, for example, mental health, screen time, etc. 
over long-term impacts.

Child-centred  methodologies  should  be 
considered;  some  cognitive  and  metacognitive 
skills  are  still  developing  in  children  and  the 
consideration of children’s developmental stage 
would benefit the studies on AI and child’s rights 
(e.g. Zaman, 2020). 

Cross-cultural  studies  would  allow  the 
identification  of  patterns  of  similarities  and 
patterns that differ among children which grow-
up in diverse cultural contexts.

Children’s  critical  reflections  would  greatly 
benefit  not  only  the  study  of  AI  and  children’s 
rights,  but  children  themselves.  This  requires 
the creation of opportunities and the support of 
children to reflect on AI in a critical way (Charisi 
et al., 2020b).

Novel short-term methods as for certain AI 
areas, policymakers need scientific evidence in 
short term. In such cases, we might need to think 
of some methods that are not used typically. For 
example,  qualitative  ethnographic  studies  that 
provide answers on the “why” of a phenomenon 
would allow getting an in-depth understanding 
of children’s behaviour in the context of AI, and 
would  provide  some  alternatives  in  long-term 
studies.

Some  of  the  questions  emerged  during  the 
workshops are the following:

▶ Participatory design: How can children partic-
ipate and be heard in the design and deploy-
ment of these AI systems?

▶ What kind of evaluation and monitoring meth-
ods are the most appropriate for children?

▶ How can we focus on the process of the de-

sign and evaluation of the system?

5252

CHAPTER 5

the  recent  development  of  AI,  “technology 
ambitions of many companies and governments 
still include little mention of children and even 
less careful attention to their actual needs and 
desires”.  Sometimes  the  discourse  of  digital 
natives  seems  to  be  accepted  uncritically  (i.e. 
children  and  young  people  are  already  ahead 
of adults online, so there is no need to address 
them  specifically).  Or  they  are  assumed  to  be 
the  sole  responsibility  of  their  parents,  who, 
after  all,  bare  the  consequences.  Or  they  are 
simply forgotten, seemingly invisible in general 
talk about “the population”. 

Considering Bronfenbrenner’s Ecological Theory 
of  the  development  of  the  child,  all  entities/
levels  (see  Fig.11)  that  constitute  the  context 
in  which  children  live  need  to  address  these 
conflicting  issues,  together  and  in  relation  one 
with each other. 

▶ Parents are traditionally the ones seeking to 
balance  and  accommodate  the  use  of  tech-
nology by children, arbitrating between provi-
sion, protection and participation rights (Diaz 
et  al.  2016).  Yet,  the  experience  with  new 
technology  is  new  to  them  and  for  this  rea-
son they lack points of reference and ask for 
guidance and support (Chaudron et al. 2017). 

▶ Schools, teachers and caregivers have taken 
up  the  challenge  in  many  instances  too,  ei-
ther with enthusiasm regarding the possibili-
ties offered, or due to the pressing needs for 
protecting children from harm, generated by 
the use of the technology. Nonetheless, their 
progress is slower than the technology itself; 
they need training and support to keep pace 
and  to  use  the  technology  in  a  meaningful 
and profitable way. 

▶  Policymakers  have  made  notable  progress 
over the last decade in considering children in 
their rights. Still, major concerns are linked to 
the technology ambitions of many companies, 
whose business model puts constant pressure 
on children’s data and attention, as it is based 
on data collection, data flow and data analy-
sis and commercial maximisation of the ret-
ro-nurture of this cycle. 

It seems urgent for the society – as a whole – 
to listen to children’s needs and desires and to 
help find a fruitful balance.

▶  How  can  we  combine  community  and  child 
needs with designs derived by companies?

▶  How  can  we  ensure  that  our  methodologies 

are human-centred?

▶  How  can  we  undertake  the  multidisciplinary 

approach in AI, in a systematic way?

Most importantly, the consideration of children 
as  one  of  the  stakeholders  and  actors  in  the 
development  of  policy  and  research  on  AI  for 
children  would  not  only  benefit  all  the  parties 
involved,  but  it  would  entail  a  significant 
learning experience for children as well.

technology  with  a 

Lastly,  we  observed  that  the  research  on  AI-
based  systems  for  children  often  has  as  its 
starting point the development and availability 
limited 
of  a  specific 
consideration  of  the  impact  of  the  integration 
of  this  technology  in  the  wider  eco-system  of 
children.  For  this  reason,  we  propose  a  critical 
approach  starting  with  questioning  existing 
problems  that  children  currently  face  and 
whether AI is a viable candidate to solve those 
problems. 

5.2.4 Balancing conflicting rights
Preparatory  work  for  this  report  highlighted 
that  various  situations 
involving  children’s 
rights in the use of AI technology may appear 
to be in conflict with one another. For example, 
adopting  restrictive  positions  on  the  use  of  AI 
by children with the aim to protect their privacy 
and  personal  data  could  impact  on  children’s 
rights  of  participation  in  the  public  debate. 
By  contrast,  using  AI  technology  to  improve 
internet  safety  among  children  would  require 
large  data  collection  and  profiling  exercises, 
putting additional pressure on children’s rights 
to  privacy  and  protection  of  personal  data. 
We  can  find  the  same  conflict  when  using  AI 
technology to personalise and tailor educational 
tools to the pupils’ needs. 

Also,  children’s  rights  might  conflict  with 
other  priorities  of  society,  such  as  the  agility 
of  industry  (e.g.  developing  safe  and  secure 
products  adds  time  and  uses  resources  in  the 
development of products). 

Not  withstanding  the  increased  attention  of 
children’s  rights,  the  situation  depicted  by 
Livingstone 
(2015),  when  considering  the 
internet  in  2015,  is  still  valid  today  regarding 

5353

CHAPTER 5

5.3. Identified knowledge gaps 
requiring further research

Pressing  knowledge-gaps  and  three  priority 
areas  for  research  have  been  stressed  by  the 
results of this research: 

I. Children’s cognition, development, and play, 

II. Empowering through education, and 

III. Age verification systems. We conclude here with 
a table displaying pressing research questions that 
need to be addressed (see Table 4).

5.3.1  Children’s  cognition,  development, 
and play  

From a research and policymaking perspective, 
one  of  the  most  fundamental  challenges  to 
understand  and  address  is  the  impact  of  AI-
based  systems  on  children’s  cognition  and 
their  socio-emotional  development.  During  the 
workshop, the topic of the role of AI and robots 
on child development emerged numerous times. 
Accordingly, the literature reviews we conducted 
and  presented  in  Section  3  (on  the  three 
examples of AI-based applications) shows that 
there  is  already  scientific  evidence  about  the 
impact (positive and negative) of AI applications 
on  children’s  development  and  well-being. 
However, the current evidence is scattered and 
has certain limitations that need to be resolved 
for research to support policymaking sufficiently 
and  effectively.In  a  similar  line,  during  the 
discussions,  it  was  evident  that  the  following 
questions  are  high  policy  priorities  for  further 
research:

▶  What  does  AI  do  to  children’s  brains?  What 
happens  when  we  hand  over  cognitive  tasks 
to AI?

▶ How do our findings regarding the impact on 
children’s  cognition  and  development  inform 
the design of AI?

Below we elaborate some of the relevant topics 
that  relate  to  the  above-mentioned  questions 
and which were discussed during the workshop 
as areas that need further research investigation 
and policy support:

Impact. As indicated by the relevant literature 
in Section 3, the design characteristics of an AI-

system as well as the context and the way it is 
used can have a positive or a negative impact 
on  children’s  cognition  and  development. 
Robots,  for  example,  have  been  shown  to 
support  the  development  of  children’s  pro-
social behaviours (Shiomi et al., 2017). But they 
can also trigger aggressive ones (Nomura et al. 
2016; Brščić et al., 2015). What induces (or not) 
such behaviours should receive more attention 
in future research. 

Similarly,  opportunities  and  risks  appeared 
in  the  literature  review  for  conversational 
agents and recommender systems. While over-
generalisations about the use of these systems 
for  children  should  be  avoided,  it  seems  that 
further  scientific  evidence  is  needed  on  the 
elements  that  constitute  child-centred  AI,  and 
the criteria that define the positive or negative 
impact of the use of AI in relation to children.

Children’s  online-offline  play.  During  the 
youth  workshop,  our  participants  commented 
on  their  experience  with  AI-based  systems 
integrated in videogames and other leisure and 
play activities. The relevant literature indicates 
that  children’s  free  play  is  essential  to  their 
development  and  growth,  and  contributes 
hugely to their well-being. However, while there 
is  an  extensive  literature  on  children’s  off-line 
play and the ways children develop in safe play 
environments with the use of their imagination 
and creativity, it is still unclear how existing AI 
systems, embedded in devices that children use 
in their play (personal assistants, smartphones, 
tablets,  video  game  consoles  or 
Internet 
transform  children’s  play 
connected 
activities  and  possibly  the  ways  they  develop 
and behave. One concrete example are children’s 
smart toys. While from a policy perspective, the 
European  Commission  is  active  in  regulating 
smart toys for children, the question is still open 
regarding  not  only  children’s  protection  during 
play but how these devices should be designed 
to facilitate children thriving.

toys) 

Use  findings  to  design  AI  better  for 
children. While understanding the impact of AI 
systems  on  children’s  cognition,  development 
and  play  is  necessary  for  the  formulation  of 
policies  regarding  AI  and  children,  this  also 
has  the  potential  to  affect  the  development 
of AI for children. Our understanding of child’s 
cognition  and  behaviour  in  the  context  of  AI, 
the  policymaking  for  AI  systems,  as  well  as 
the  actual  AI  design  and  development  of  AI 
systems are all interrelated. While some of the 

5454

CHAPTER 5

connections between the elements are already 
emerging, such as children’s participation in the 
design of AI-based technology or recently in the 
policymaking process, the connections and the 
interplay  among  these  three  dimensions  still 
need to be supported further.

5.3.2 Empowering through education
As  discussed  in  Section  3,  the  use  of  AI  and 
robotics  technology  may  have  an 
impact 
on  children’s  well-being  as  well  as  on  their 
development,  especially  cognitive  and  socio-
emotional  aspects.  This  report  looks  at  these 
two  interconnected  issues  through  the  lens 
of  children’s  rights.  To  both  protect  children, 
but  also  empower  them  to  take  advantages 
of  these  modern  technologies  to  participate 
in  society  and  for  young  persons  to  enter  the 
labour  market  successfully,  two  separate  sets 
of  competences  stand  out  from  the  common 
European  reference  for  Key  Competences  for 
Life, namely digital competences and personal, 
social and learning to learn competences.⁶⁴

We  take  a  brief  look  at  how  policymakers, 
education  and  training  providers,  as  well  as 
social partners could use this common reference 
framework  to  guide  their  implementations, 
in  order  to  leverage  AI  systems  for  children’s 
education,  health  care  and  right  to  play,  while 
upholding  children’s  rights  and  empowering 
their  agency.  Using  a  common  reference 
framework helps to create an agreed vision and 
a  collective  understanding  through  a  common 
vocabulary – a linga franca. This helps mobilise 
stakeholders  to  take  initiatives  and  to  learn 
from each other.⁶⁵  

Following  the  Council’s  Recommendation  on 
Key  Competences  for  Life  (mentioned  above), 
the Digital Competence Framework for Citizens 
(DigComp)  defines  the  knowledge,  skills,  and 
attitudes  that  citizens  need  to  engage  with 
digital  technologies  in  a  confident,  critical, 
and responsible way for learning, at work, and 

for  participation  in  society.⁶⁶  The  European 
Commission  first  published 
the  DigComp 
framework  in  2013,  and  today,  many  of  the 
European  education  systems  refer  to  the 
European key competence definition for digital 
competence, in their report on digital education, 
Eurydice (2020).⁶⁷ 

DigComp is also referenced in the new Recovery 
and  Resilience  Plans  for  developing  human 
capital through digital upskilling. 

The  DigComp  update  2.2,  released  in  spring 
2022,  adds  new  aspects  by  considering 
competences that citizens need when interacting 
with  AI  systems.  These  include  aspects  such 
as 
like 
interacting  with  virtual  assistants 
conversational agents and chatbots, while being 
cognisant  about  sharing  personal  data  (e.g. 
one’s voice and facial image) and taking steps 
to  safeguard  them  from  being  overly  shared 
and  processed  (e.g.  sensibility  towards  digital 
data,  protection,  and  privacy).  Equally,  the 
update  will  consider  examples  of  knowledge, 
skills  and  attitudes  related  to  creating  digital 
content  and  non-digital  artefacts  such  as 
tinkering with sensors and robotics in a way that 
it can benefit STEAM activities. Such examples 
can  help  education  systems  further  consider, 
for  example,  makerspace  activities  as  part  of 
STEM  education  which  can  be  conducive  for 
developing a wider array of cognitive processes 
and  strategies  such  as  creative  thinking, 
reasoning and creativity.⁶⁸  

it 

Therefore,  the  DigComp  2.2  update  can  be 
considered as an important tool for empowering 
children  in  compulsory  education  (ISCED  1-3), 
as 
is  well-known  that  national/regional 
curriculums  are  often  the  place  where  values 
related to education equity are translated into 
action.  By  the  way  of  offering  a  gateway  for 
education  systems  to  consider  including  these 
new aspects in their curriculums, the DigComp 
update 2.2 can play a significant role in creating 
a more equitable education system. 

NOTES
⁶⁴ https://ec.europa.eu/education/education-in-the-eu/council-recommendation-on-key-competences-for-lifelong-learning_en

⁶⁵ See for example ”The use of reference frameworks to support digitally competent citizens – the case of DigComp”:  
    https://academy.itu.int/sites/default/files/media2/file/Digital%20Skills%20Insights%202019%20ITU%20Academy.pdf

⁶⁶ https://ec.europa.eu/jrc/digcomp 

⁶⁷ https://eacea.ec.europa.eu/national-policies/eurydice/content/digital-education-school-europe_en 

⁶⁸ Makerspaces for Education and Training: Exploring future implications for Europe:  
    https://publications.jrc.ec.europa.eu/repository/handle/JRC117481

5555

CHAPTER 5

In  general,  AI  applications  and  systems  could 
gain from focusing more on children’s well-being. 
For example, placing more focus on aspects that 
engage  and  foster  children’s  personal,  social 
and  emotional  learning  could  help  developing 
AI applications that reinforce competences such 
as  taking  agency  and  initiative,  perseverance 
intellectual  openness,  self-regulation, 
and 
negotiation  and  conflict  resolution  skills.⁶⁹  
Some  of  such  competences  are  also  part  of 
the  LifeComp  framework  developed  by  the 
European Commission in 2020.⁷⁰

The  LifeComp  framework  can  be  used  as  a 
basis for curriculum development and learning 
activities  (a  repository  of  teaching  practices 
will  be  published  in  spring  2022).⁷¹  This  is  a 
way  of  empowering  young  users  of  AI-driven 
systems,  which  sometimes  are  designed  to 
circumvent  user’s  own  will  through  nudging, 
gamification  and  manipulation,  in  order  to 
influence  behaviour:  for  example,  to  continue 
watching  videos  or  interacting  through  chats, 
as mentioned in previous sections.

5.3.3 Developmentally appropriate 
systems and age verification 

Easy  access  to  the  online  world  has  provided 
certain  opportunities  for  children,  yet  it  also 
poses  severe  threats  to  their  protection.  AI 
systems should be developmentally appropriate 
and adapt their content and the communication 
styles  according  to  the  children’s  cognitive 
and  socio-emotional  levels.  In  addition,  AI 
age  verification  tools  have  been  explored  to 
protect  children  online.  These  tools  are  used 
to ensure people do indeed show the required 
age  to  sign  up  to  certain  platforms.  Meta  (the 
umbrella  company  for  Facebook,  Instagram 
and Whatsapp) states that it is using AI as “the 
cornerstone  approach”  to  estimate  its  users’ 
age.⁷²  However,  experts  emphasise  that  age 

in  itself  does  not  entail  strict  correspondence 
with  the  competences,  capacities,  the  agency 
or  the  resilience  of  a  child.  Other  elements, 
such as levels of cognition, of (digital) literacy, 
context and background in which he/she grows 
matter greatly, and can lead to varied skills and 
capacities  among  children  sharing  the  same 
age. 

important  questions  emerged 

Further 
in 
the  experts'  discussions:  how  can  the  legal 
obligation  of  privacy-by-design  take  children’s 
views  to  safeguard  their  privacy?  How  can 
age  verification  tools  powered  by  AI  be  used 
in  full  compliance  with  the  right  to  privacy 
and  EU  legislation?  International  organisations 
and foundations such as IEEE and the 5Rights 
foundation  have  already  done  considerable 
work  along  these  lines,  by  publishing  a  new 
standard⁷³  on  the  age-appropriate  verification 
of AI systems for children.⁷⁴

5.3.4 Other research questions to be 
addressed

During  the  workshops  we  requested  the 
participant  experts  and  policymakers 
to 
contribute questions relevant to AI and children’s 
rights  that  they  would  prioritise  for  a  future 
integrated  policy  and  research  agenda.  We 
received  50  questions  in  total.  However,  when 
examining and commenting on the questions, we 
observed that they tackled topics that could be 
grouped and abstracted into categories. For this 
reason, we performed a thematic analysis which 
resulted in the categorisation of the questions 
into  7  groups:  Inclusion,  Education,  Privacy, 
Explainability-Transparency-Accountability, 
Cognition-Development,  Conflicting  Rights  and 
Evaluation-Monitoring.  Table  4  presents  the 
merged  categories  with  the  corresponding 
questions.

NOTES
⁶⁹ Emerging technologies and the teaching profession (2020): https://publications.jrc.ec.europa.eu/repository/handle/JRC120183

⁷⁰ https://ec.europa.eu/jrc/lifecomp 

⁷¹ A self-paced MOOC for teachers: https://www.schooleducationgateway.eu/en/pub/teacher_academy/catalogue/detail.
cfm?id=232029&cbmid=45658275 

⁷² How Facebook Knows an App User Is Old Enough | Meta (fb.com), https://about.fb.com/news/2021/07/age-verification/, accessed on 
02/02/2022

⁷³ https://standards.ieee.org/standard/2089-2021.html

⁷⁴ https://5rightsfoundation.com/static/ieee-2089-2021.pdf?_cchid=edb63f689841fb8cfba977a86edf49c3 

5656

CHAPTER 5

TABLE 4
Indicative questions generated during the Experts’ workshops
Source: EC

Inclusion

Education

Privacy

Disproportionate collection of data 
to ensure the inclusion of 
underrepresented groups of 
children ?

How can we build the agency of 
students (K-12) to be informed, 
empowered citizens and users of 
AI? 

What is 'fair ' processing of 
children's personal data?

What are and how can we 
develop the self-protection skills 
that children need to develop for 
a healthy use of AI systems (e.g. 
social media), including digital 
skills but also socio-emotional 
skills?

How can bias and lacking 
representativity in data sets used 
for machine learning be addressed 
to avoid discriminatory outcomes?

How can we ensure that the 
development of AI apps considers 
children from different contexts 
and backgrounds, different 
socio-economic status and ethnic 
identity?

How do the specific context (e.g. 
those more vulnerable) may lead 
to different ethical impacts 
(dilemmas) in the use of AI 
systems?

How do we measure inclusion? 

Cognition / 
development

Conflicting rights

Evaluation / 
monitoring

Explainability/
transparency/
accountability 

Who are the stakeholders that 
should be involved, given a 
particular use case? 

What kind of partnerships are 
needed in the long term? 

Should, and if so how, AI systems 
embed educational features (as 
"teachable moments" favoring 
transparency and skill 
development) being triggered 
when needed in an open use of AI 
systems?

How to conciliate inclusion needs 
(more data collection) and data 
protection - privacy? 

Are users aware of the presence 
of AI technology on the online 
platform they decide to visit

Are some rights more important 
than others for children in relation 
to AI? What is the framework for 
evaluating/balancing them? AND 
how to deal with conflicting 
rights? (e.g. protection - autonomy

When we feed the AI with data, 
are the users well informed about 
which data it is and for what 
purpose will it be used?

Are children (potentially all users) 
informed about it in a clear and 
easy-to-understand way?

What are children expected to 
understand at different 
developmental stages. 
What is the impact of specific 
ai-based technologies on 
children's cognitive and 
socio-emotional development?

What are the cognitive and 
socio-emotional processes 
resulting from the use of AI 
systems that advance or inhibit 
children well-being? 

Impact of greater integration of 
datasets (single digital identity, 
etc.) on rights of children, beyond 
general data protection. Example: 
incentivisation of behavioural 
change in one environment with 
rewards in another.

How should we design hybrid 
AI-human systems that hand over 
the right cognitive tasks to AI vs. 
human, given effects on children 
development and agency? Should 
the behaviour of these systems 
be adaptive towards less AI 
support

How the changes in the ways we 
access the internet (from screens 
to environmental access through 
IoTs and voice-based assistants) 
change our knowledge of the 
world (our epistmeology)?

5757

 
5858

CHAPTER 6

6. RESEARCHERS, POLICYMAKERS 
AND CHILDREN: A TRIANGULATION OF 
PERSPECTIVES

In this section, we elaborate on a set of topics and questions that can indicate the future 

directions for research and policy. These topics are the results of questions and feedback 
received by policymakers, experts, researchers, and young participants that we involved in 

our workshops.

We are interested in understanding the connections among the perspectives of these three 
types of stakeholders in terms of priorities for future directions. We performed a thematic 
content analysis which resulted in the identification of 5 basic topics that were discussed 
across stakeholders. We also annotated additional topics that were discussed from a single 
stakeholder as well.

As illustrated in the figures below, this analysis reveals that while children and policymakers 
emphasised education, researchers emphasised more children’s cognition and development 
in relation to AI. 

In addition, inclusion appeared to be a topic of priority for policymakers and researchers while 
for children this did not appear as a priority. Interestingly, transparency and explainability 
seem to be important for researchers and children while for policymakers this was the topic 
given the least priority of all. The reason for these differences might be identified by the fact 
that topics that are still in their infancy from a research point of view (such as explainability) 
are of less interest to policymakers, whereas topics that can already demonstrate research 
findings (such as education and privacy) are more interesting for policymakers while being 
less attractive from a research point of view.

These findings indicate it is important to consider that different stakeholders have different 
priorities for the development of an integrated agenda. This should be communicated among 
stakeholders in a transparent way and in future agendas to consider their harmonisation.

5959

CHAPTER 6

In  this  section,  we  present  the  comparison  of 
the  topics  mentioned  by  all  the  three  types 
of  stakeholders  (children,  policymakers,  and 
researchers)  while  considering  AI  technology 
and  discussing  future  directions  for  research 
and policymaking. For us to clarify how different 
stakeholders  prioritise  different  topics  for 
further  research  and  policy,  we  performed  a 
thematic content analysis of their contributions 
to our workshops. These are summarised by the 
following topics: 

1. Questions that we received from invited poli-
cymakers from the European Commission and 
other policy-oriented institutions (n=28 ques-
tions);

2. Questions  that  emerged  during  the  series 
of  three  workshops  with  expert  researchers 
(n=22 questions);

3. Topics that emerged during the series of two 
workshops  with  children  and  young  people 
(n=35 topics).

Figure 7 shows that the questions we received 
by  the  Policy  DGs  and  bodies,  emphasised 
mainly  children’s  inclusion,  education  and  on 
the  impact  of  AI  on  children’s  cognition  and 
development.  In addition, children’s exploitation 
is a topic of interest from a policy point of view, 
in addition to privacy and explainability.

of  systems  was  put  forward  by  researchers 
but  not  by  policymakers.  The  reason  for  these 
differences  might  be  identified  by  the  fact 
that topics that are still in their infancy from a 
research  point  of  view  (such  as  explainability) 
are  of  less  interest  to  policymakers,  whereas 
topics  that  can  already  demonstrate  research 
findings  (such  as  education  and  privacy)  are 
more  interesting  for  policymakers  while  being 
less attractive from a research point of view.

the 

results  drawn 

Lastly,  we  contrasted  the  above-mentioned 
perspectives  of  policymakers  and  researchers 
with 
from  children’s 
perspectives  (Fig.9).  As  expected,  for  children, 
education and AI literacy are priorities while they 
are curious about the evaluation and monitoring 
of  the  AI  systems  that  are  being  designed  for 
them.  Interestingly,  privacy  is  among  of  the 
priorities  which  for  children  appear  as  more 
important  than  it  does  for  policymakers  and 
researchers. 

Children  are  also  interested  in  understanding 
better the development of explainable systems 
for them and they introduced topics that were 
not discussed with the other two stakeholders, 
such  as  the  impact  of  AI  on  employment.  The 
topic of employment might appear as a priority 
in the results of this workshop because of the 
age-group of our participants who were in late 
adolescence.

Regarding  the  questions  that  emerged  in  the 
discussions  among 
invited  experts  during 
the  workshops  (Fig.8),  we  observe  that  an 
emphasis  was  placed  on  children’s  inclusion 
and to the impact of AI on children’s cognition 
and  development.  The  topics  of  explainability 
and  evaluation/monitoring  of  AI  systems  are 
among  the  topics  researchers  believe  should 
be given priority as well. By contrast, education 
and privacy were not classified among the top 
priorities  from  a  research  point  of  view.  The 
question of conflicting rights was an additional 
topic emerged during the discussions.

To  summarise,  this  analysis  revealed  some 
patterns  in  terms  of  the  prioritisation  of 
future  directions  from  three  stakeholders, 
It 
policymakers,  researchers,  and  children. 
appears  that  all  the  stakeholders  introduce 
several similar topics for examination; however, 
it  seems  that  the  priorities  might  differ.  We 
acknowledge that these results are based on a 
very small sample of participants in all the three 
categories  (policymakers  N=28,  researchers 
N=22,  children  N=35)  and  they  are  biased  by 
the  participants’  background,  speciality  and 
interests. 

If we examine the perspectives of researchers 
(Fig.8)  compared  to  those  of  policymakers 
(Fig.7),  we  can  conclude  that  for  both  groups, 
children’s  inclusion  for  the  development  and 
use  of  AI  and  the  impact  of  AI  on  children’s 
cognition  and  development  are  among  the 
priorities. Interestingly, although education and 
privacy appear to be a priority for policymakers, 
it seems that they receive less attention from a 
research point of view, while the explainability 

However, these findings provide initial indications 
for  directions  of  future  work  and  the  need  to 
develop  structures  and  frameworks  for  the 
systematic combination of the perspectives of 
various stakeholders who might have different 
priorities in terms of AI for children, as well as 
for systematic ways of communication among 
stakeholders.

6060

CHAPTER 6

FIGURE 7
Questions received from policymakers (N=28)
Source: EC

Exploitation
14.3%

Exploitation
Cognition / Development
14.3%
21.4%
Exploitation
14.3%
Cognition / Development
Explainability
21.4%
3.6%
Privacy
Cognition / Development
10.7%
21.4%
Explainability
3.6%
Privacy
Explainability
10.7%
FIGURE 8
Employment
3.6%
Questions emerged through the discussions with experts (N=22)
Privacy
2.9%
Source: EC
10.7%
Evaluation / Monitoring
Employment
17.1%
2.9%
Conflicting rights
Employment
8.6%
Evaluation / Monitoring
2.9%
17.1%
Cognition / Development
Evaluation / Monitoring
11.4%
Conflicting rights
17.1%
8.6%
Privacy
Conflicting rights
Cognition / Development
11.4%
8.6%
11.4%

Cognition / Development
Privacy
11.4%
11.4%

Privacy
Evaluation
11.4%
13.6%
FIGURE 9
Topics posed by children and youth (N=35)
Source: EC

Evaluation
Conflicting rights
13.6%
9.1%
Evaluation
13.6%
Conflicting rights
Cognition / Development
9.1%
27.3%
Conflicting rights
9.1%
Cognition / Development
27.3%

Cognition / Development
27.3%

Inclusion
25%

Inclusion
25%

Inclusion
25%
Education
25%

Education
25%

Education
25%
Trust
8.6%

Trust
Education
8.6%
22.9%
Trust
8.6%

Education
Privacy
22.9%
17.1%
Education
22.9%
Privacy
17.1%

Privacy
17.1%
Inclusion
22.7%

Inclusion
22.7%
Education
9.1%
Inclusion
22.7%
Privacy
Education
4.5%
9.1%
Explainability
Education
13.6%
Privacy
9.1%
4.5%

Explainability
Privacy
13.6%
4.5%

Explainability
13.6%

6161

6262

CHAPTER 7
CHAPTER 5

7.  LIMITATIONS AND CONSIDERATIONS

We  acknowledge  that  this  report  and  the 
certain 
workshops 
limitations.

come  with 

included 

▶ First, it was beyond the scope of this report 
to consider an exhaustive and systematic re-
view  of  all  existing  policy  initiatives  and  the 
AI-based applications for children. Instead, we 
decided  to  make  a  representative  selec-
tion of them, being aware that other impor-
tant work is not included in the report. 

▶  The  results  of  the  workshops  with  children 
and experts were based on a selection of a 
small number of participants; we acknowl-
edge  that  the  outcome  of  the  discussions  
might  be  biased,  given  the  background  of 
the participants. However, we tried to include 
participants that represent a variety of disci-
plines, sectors as well as geographical areas, 
in order to achieve increased diversity in the 
group of participants.

▶ The young people in the workshop were ado-
lescents; children of younger age-groups and 
their  insights  and  needs  are  not  fully  repre-
sented  in  our  findings.  We  acknowledge  that 
there are large developmental differences 
among  different  age-groups  of  childhood 
and adolescence which should be considered 
in future work.

▶ The participants of the workshops came from 
various  backgrounds  and  locations.  Yet,  they 
provided a Northern and Western perspec-
tive  of  a  topic  that  spans  the  world’s  pop-
ulation.  It  was  beyond  the  scope  of  this  re-
port  to  consider  populations  outside  Europe. 
However, we are aware that developments in 
the EU cannot be seen in isolation; rather, they 
need to be placed in the context of the glob-
al developments, in North America, China and 
the emerging opportunities and challenges in 
developing countries, such as in sub-Saharan 
Africa. Decisions taken in the EU are inextrica-
bly linked to developments at a global level.

▶ Lastly, it has been beyond the scope of this 
report  to  include  perspectives  of  indus-
try,  during  the  workshops  with  experts  and 
policymakers, it was made clear that the in-

volvement  of  industry  should  be  considered 
and  reviewed  as  an  additional  dimension  in 
the  future.  This  requires  coordinated  actions 
that would ensure the best interests of chil-
dren  with  well-documented  and  transparent 
procedures.

Based  on  the  results  of  this  report,  we  will 
continue our work considering the requirements, 
the methods and the areas of knowledge-gaps 
in research. It is important to meet these gaps 
in  the  current  policymaking  agenda.  For  our 
future  work,  we  will  consider  the  following 
prerequisites:

▶  Policy  and  research  on  artificial  intelligence 
and  children  need  to  consider  the  best  in-
terest of children as indicated in the UN 
Convention on the Rights of the Child, in 
the context of the current and future techni-
cal development of AI. The relevant regula-
tory  frameworks  for  AI-based  applica-
tions that are meant to be used by children 
or  might  impact  children’s  well-being  in  the 
short or long term should prioritise children’s 
best interests and benefit the research & de-
velopment of AI for children.

▶  The  pervasiveness  of  AI-based  technologies 
in children’s lives calls for strategic planning 
for a joint coordinated approach to their 
governance  through  international  coop-
eration. Regulatory frameworks could be de-
signed in a way that fosters innovation in line 
with EU values and standards set to leverage 
the EU’s ability for leadership.  

▶ Policy decisions need to be made in the con-
text of the global technical AI-specific ad-
vancements and the corresponding ethi-
cal perspectives. 

▶ Policy decisions need to consider not only the 
current financial and societal impact but also 
the  dynamic  long-term  impact  that  these 
decisions have in future societies via the di-
rections of children’s development.

▶  Multistakeholder  collaboration  requires  the 
strategic  planning  of  coordinated  ac-

6363

 
 
CHAPTER 7

tions with an agreed threshold of values that 
prioritise  children’s  best  interests  and  that 
ensure  a  balanced  combination  with  special 
attention paid to possible conflicting areas of 
application.

▶  Involvement  of  industry.  The  evaluation 
of  AI-based  tools  for  children  often  needs 
real-world scenarios. This means that indus-
try should be considered as one of the actors 
that play a significant role in our further un-
derstanding  and  decision-making  regarding 
AI and children. During the workshop discus-
sions,  however,  many  concerns  were  raised 
regarding  possible  conflicting  agendas  in 
the  interaction  of  industry  with  researchers 
and policymakers. For this reason, the group 
highlighted  the  need  for  developing  and  es-
tablishing  frameworks  and  structures  that 
would  ensure  transparent  interactions  and 
collaborations  among  different  stakeholders, 
including industry. Most importantly, the par-
ticipants of the workshop focused on the need 
for defining and agreeing upon a set of com-
mon values and goals regarding the develop-
ment and use of AI for children.

In the future, the JRC aims to create an open-
access  pool  of  best  practices  for  projects  that 
integrate  science  and  policymaking,  with  the 
active  participation  of  children.  The  main  goal 
of  this  pool  is  to  develop  further  the  existing 
tools and strategies and to encourage the use 
of them by stakeholders that are involved in the 
development of AI for children.

6464

CHAPTER 8

really necessary to accomplish a valuable pur-
pose. Despite relevant research in the domain, 
even  more  research  is  needed  to  determine 
the “valuable purposes” of AI for children.

▶  AI  technology  shows  potential  to  address 
some  challenges  in  several  fields,  including 
environmental ones. Still, data centres, which 
are  critical  for  storing  the  large  amounts  of 
data needed to power AI systems, demand a 
huge amount of energy and are already point-
ed to as important contributors to increased 
CO2 emissions. Experts and the literature now 
recommend  measures  to  minimise  the  foot-
print of digital products and services, and AI 
technology  is  no  exception.  Considering  the 
negative environmental impact and the pos-
itive implications of AI for climate change will 
be crucial moving forward to better children’s 
lives.

Inclusion, non-discrimination and 
algorithmic bias 

There is a need for child-friendly AI technology 
and  for  ensuring  that  AI  systems  do  not 
systematically reflect discriminatory biases due 
to the quality of the data and algorithms they 
rely on.   

▶ Risks of exclusion and discrimination due to 
the use of ‘data biased’ AI have been the sub-
ject of extensive debates worldwide, and the 
case of children among the users has raised 
special attention. Developing a socio-ecologi-
cal approach that considers at least four levels 
of  interactions  (individual/child,  community, 
societal and policy levels) and their diversity, 
interplay,  and  relations,  seems  promising,  in 
order to enhance children’s rights to inclusion 
and  non-discrimination,  by  fostering  capaci-
ty-building and knowledge sharing.

▶ AI technology developers and designers should 
be  encouraged  and  supported  to  integrate 
children’s  rights  dimensions  in  risks  and 
impact  assessments,  evaluation  frameworks 
and guidelines. 

8.  CONCLUSIONS

This  Science  for  Policy  report  presents  the 
results  of  a  series  of  activities  on  artificial 
intelligence  (AI)  and  children’s  rights,  with  a 
focus  on  emerging  issues,  considering  the 
following  perspectives.  First,  it  provides  an 
overview  of  the  current  policy  discussions 
on  AI  and  children’s  rights  initiated  by  major 
international organisations. 

It  elaborates  on  a  selected  set  of  examples 
based on the current state-of-the-art regarding 
the  design,  development  and  use  of  three 
AI-based  applications  for  children,  namely 
recommender  systems,  conversational  agents 
and  robotic  systems.  It  then  reflects  on  the 
emerging opportunities and risks in relation to 
children’s rights. 

The  report  goes  on  to  consider  the  results 
of  a  series  of  workshops  with  children  and 
youth,  experts,  researchers  and  policymakers 
to  identify  a  set  of  requirements  for  child-
centred  AI-based  systems,  and  methods  in 
order  to  address  these  requirements,  as  well 
as knowledge gaps requiring further research 
and an examination from a policy perspective. 

Furthermore,  we  triangulated  the  perceptions 
of  the  stakeholders  involved  in  our  workshops 
(policymakers,  researchers  and  young  peo-
ple, who are legally children). We performed a 
thematic content analysis on their contribution 
which clarified how different stakeholders prior-
itise different topics in view of further research 
and policy.

REQUIREMENTS

AI minimisation, valuable purpose, 
sustainability and environmental issues 

There  is  a  need  to  consider  the  use  of  AI 
technology as a limited resource and a need for 
strategic  and  systemic  choices  to  develop  AI-
based  services,  both  at  the  public  and  private 
level. 

▶ In parallel to the concept of “data minimisa-
tion”,  an  approach  to  AI  minimisation  would 
limit the use of AI to complete tasks that are 

6565

 
 
 
CHAPTER 8

Privacy, data protection, safety

Respecting children's right for agency 

There  is  a  need  for  children  to  be  empowered 
to control how their personal data is used by AI 
technology and what the implications are. 

There  is  a  need  for  further  research  on  how 
agency  is  developed  in  children  when  they 
interact with AI-based systems. 

▶  Considering  its  opaqueness  and  complexity, 
the specific nature of AI poses additional risks 
on top of those already relating to privacy and 
data protection. There must be an effort – as 
mandated  by  the  GDPR  –  to  make  privacy 
statements  understandable  by  children, 
and  support  them  in  exercising  their  data 
protection rights. 

▶ Children’s right to agency is one of the funda-
mental requirements to be met when design-
ing and using AI for children: allowing children 
to  be  consciously  agents  and  actors  within 
their interaction with AI technology. Research, 
while  recognising  children’s  agency  to  be  a 
complex  process,  has  so  far  only  considered 
children’s perceived agency.

▶ Data protection principles, such as privacy-by-
design, should take children into consideration 
and  translate  into  effective  mechanisms  to 
ensure their protection.

Transparency, explainability, accountability 

There is a need for transparency, explainability 
and  accountability  to  inform  and  empower 
young  citizens  and  all  users  of  AI  technology, 
preventing over-trust or mistrust in AI systems.

▶ Transparency and explanability are related to 
the  right  of  children  to  information  but  also 
linked  to  their  right  to  education  and  litera-
cy. Both are judged essential to promote chil-
dren’s empowerment, critical thinking and to 
fight misinformation. 

▶ Reaching transparency and explainability re-
quires  the  involvement  and  collaboration  of 
all  relevant  stakeholders  (children,  research-
ers,  policy  makers,  civil  society,  industry  and 
parents). 

▶ Along with performance and accuracy, trans-
parency  and  explainability  have  become  ad-
ditional  criteria  and  a  priority  for  evaluating 
algorithms. 

▶ Accountability is essential, at an organisation-
al  and  system  level,  in  auditing  procedures. 
Still,  AI  systems  pose  unique  challenges  as 
their  input  and  operations  are  not  always 
visible,  especially  to  the  end-users.  For  AI 
systems  that  are  designed  for  children,  par-
ticularly,  designers  must  identify  minimum 
requirements that would ensure and monitor 
the accountability of the actors involved.

METHODS

Anticipation, evaluation and monitoring

There is a need to create frameworks and toolkits 
that can enable/guide the design and evaluation 
of  AI  systems  in  the  short-  and  long-term, 
having children as their main users or including 
them  among  users.  These  frameworks  should 
incorporate aspects such as data protection and 
risk assessment. 

▶  Research  dedicated  to  the  evaluation  and 
monitoring  AI  technology  used  by/designed 
for children are most of the time rather limit-
ed in terms of case-studies, stakeholders and 
time frames. 

▶  Literature  and  experts  participating  in  this 
work  agreed  that  children-centric  evaluation 
should  expand  and  combine  different  meth-
odologies,  including  participatory  methods, 
interdisciplinary  research,  collaborative  con-
sultation  of  stakeholders  and  large-scale 
studies. It should also consider different time 
scales: short, medium and long-term studies, 
the latter being the implemented most rarely.

Multi-stakeholder collaboration

There 
is  a  need  of  communication  and 
collaboration  between  stakeholders  to  seek 
balance in the use of AI technology by children 
and resolve the conflicts between their provision, 
protection and participation rights. 

▶  Each  stakeholder  group  (children,  parents, 
teachers,  carers,  academia,  industry  and 
policymakers)  has  specific  aims  and  goals 
in  using/developing  AI 
technology  and 
supporting  children’s  rights.  Communication 

6666

CHAPTER 8

and collaboration are key to defining common 
goals  and  to  building  child-friendly  AI  by 
design. 

▶  The  future  agenda  for  AI  technology  should 
include  the  outline  of  a  simple  framework 
(or  guiding  questions)  that  would  facilitate 
communication among stakeholders. 

▶  Platforms  that  would  support  and  facilitate 
the interaction and collaboration of different 
stakeholders, including children and industry, 
should be developed further.

Children’s participation 

Children’s participation, provision, and protection 
(the 3 Ps model) are equally important. Yet it is 
challenging  to  meet  these  fully,  and  all  policy 
decisions need to ensure that the 3 P’s model is 
performed in a timely and meaningful manner, 
by  taking  into  consideration  the  cognitive  and 
socio-emotional  stage  of  each  child  and  its 
fundamental right to development through play.

▶ Scientific evidence requires innovative child-
centred methods that consider children’s rapid 
development, the importance of context and 
the local value-ecosystems in which children 
develop, as well as AI-specific characteristics 
such as adaptation.

▶  The  different  characteristics  of  applications 
require  a  fine-tuned  approach 
indicating 
that we cannot have one solution that fits all 
applications  within  the  various  contexts  and 
for different users. We need principles and best 
practices  that  provide  directions  for  multiple 
characteristics  of  AI-based  applications  for 
children and their different uses, in accordance 
with children’s characteristics.

▶ Children’s meaningful participation and their 
consideration  as  core  stakeholders  can  be 
the means to in-depth understanding of their 
perspectives, needs and interests. At the same 
time,  this  process  is  important  for  children’s 
holistic grasping of the process of AI systems 
which would allow them to make connections 
and create personal meanings.

Conflicting rights

There  is  a  need  for  regulation  and  policies 
on  AI  technology  to  consider  the  full  range 
of  children’s  rights  (not  only  protection),  and 

to  define  the  correct  degree  of  compromise 
between different, sometimes conflicting rights.

▶  According  to  the  United  Nations  Convention 
of  the  Rights  of  the  Child,  rights  can  be 
three  areas,  namely: 
into 
categorised 
protection, 
provision. 
participation  and 
Conflicts among these rights are not rare. For 
instance, adopting restrictive positions on the 
use of AI by children with the aim of protecting 
their privacy and personal data could impact 
on  children’s  rights  in  participating  public 
debate.  By  contrast,  using  AI  technology  to 
improve Internet safety among children would 
require  large  data  collection  and  profiling 
exercises,  putting  additional  pressure  on 
children’s  rights  to  privacy  and  protection  of 
personal data.

▶  Decisions  concerning  the  correct  degree  of 
compromise  demand  careful  consideration 
of  trade-offs  between  different,  sometimes 
conflicting rights. 

▶  Also,  children’s  rights  might  conflict  with 
other priorities of society, such as the agility 
of  industry  (i.e.  developing  safe  and  secure 
products adds time and the use of resources 
in the development process of products).

▶ Regulation and policies on artificial intelligence 
should  consider  the  full  range  of  children’s 
rights,  especially  when  these  are  conflicting. 
Likewise,  industry  would  need  to  take  the 
necessary time and care to develop safe tools 
respectful of children’s rights by design.

KNOWLEDGE GAPS

Cognition, development and children’s play 

There is a need to study deeper the impact of 
the use of AI technology, embodied or not, on 
children’s behaviour, the brain, as well as the 
cognitive and socio-emotional capacities of 
children.  

There  is  already  scientific  evidence  about 
the  impact  (positive  and  negative)  of  AI-
based  applications  on  children’s  cognition  and 
development.  However,  the  current  evidence 
is  scattered  and  has  certain 
limitations. 
Understanding  and  addressing  the  impact  of 
AI-based  systems  on  children’s  cognition  and 
their socio-emotional development is one of the 
most fundamental challenges from a research 

6767

 
 
and the background in which a child grows up 
matter  greatly  and  can  lead  to  varied  skills 
and  capacities  among  children  of  the  same 
age. 

TRIANGULATION OF THE 
RESULTS: DIFFERENT 
PRIORITIES BY DIFFERENT 
STAKEHOLDERS

While most of the above-mentioned topics were 
mentioned by all the three types of stakeholders 
involved in our workshops (children, policymak-
ers and researchers) during the discussions, we 
observed that different stakeholders emphasise 
and prioritise different topics. 

This was confirmed by their written contributions 
and  the  questions  they  proposed  on  AI  and 
children’s rights. In order to clarify how different 
stakeholders  prioritise  different  topics  for 
further  research  and  policy,  we  performed  a 
thematic content analysis of their contributions. 

As  illustrated  by  the  figures  in  Section  6,  this 
analysis  reveals  that  while  children  and  poli-
cymakers  emphasised  education,  researchers 
emphasised  more  children’s  cognition  and  de-
velopment in relation to AI. In addition, inclusion 
appeared to be a topic of priority for policymak-
ers and researchers while it did not appear as a 
priority for children. Interestingly, transparency 
and explainability seem to be important for re-
searchers and children.

These findings indicate that for the development 
of  an  integrated  agenda,  it  is  important  to 
consider 
that  different  stakeholders  have 
different priorities. This should be communicated 
among stakeholders in a transparent way, with 
a view to harmonising future agendas.

CHAPTER 8

and  policymaking  perspective.  The  following 
questions  appear  to  be  of  high  policy  priority 
for further research: 

▶ What does AI do children’s brains? What hap-
pens when we hand over cognitive tasks to AI, 
in a positive and in a negative way?

▶ How do AI systems transform children’s play 
activities,  imagination  and  creativity,  and 
eventually the ways they develop and behave?

▶ How do our findings regarding the impact on 
child’s cognition and development inform the 
design of AI?

Provision from the perspective of education 

There  is  a  need  to  develop  competences  and 
literacy  that  will  prepare  children  for  a  world 
transformed by AI technology but also to develop 
AI  that  can  develop  competences  supporting 
children’s well-being. 

▶  Young  people  show  concerns  regarding  the 
future of their education and work in a world 
transformed  by  AI  technology.  They  request 
schools to develop adequate digital and tech-
nical  competences,  but  also  critical  and  eth-
ical  thinking  as  well  as  personal,  social  and 
self-learning competences.

▶ AI applications and systems could gain a lot 
from  focusing  on  children’s  well-being.  For 
example,  more  emphasis  could  be  placed  in 
developing AI applications that reinforce com-
petences,  such  as:  taking  agency  and  initia-
tive; perseverance and intellectual openness; 
self-regulation; negotiation and conflict reso-
lution skills. 

Age verification systems while using AI 

There is a need to expand further age recognition 
tools so as to consider the large variety of skills, 
capacities and agency levels of children sharing 
the  same  age,  as  well  as  children's  views  on 
how to safeguard their privacy. 

▶  Age  verification  tools  have  been  considered 
to  ensure  the  protection  of  children  online 
and keep them away from content or services 
that are not meant for them. However, focus-
ing only on age might lead to a false sense of 
security or, on the contrary, limit children’s op-
portunities. Other elements, such as the level 
of cognition, of (digital) literacy, social context 

6868

 
 
CHAPTER 9

9.  FOR DEEPER CONSIDERATION - 
REFLECTIONS OF INVITED EXPERTS

based on their expertise, namely (in alphabetic order): 

In this section we report reflections by the invited experts on AI and child’s rights workshop 

”Regulation and policies on artificial intelligence should consider the full range of 

children’s rights, especially when these conflict”

Victoria Baines, Bournemouth University, UK

“AI education: Beyond mere transfer of knowledge”

Tara Chklovski, Technovation, USA

“Directions  for  the  responsible  design  and  use  of  AI  by  children  and  their 
communities: Examples in the field of Education”

Davinia Hernández-Leo, Universitat Pompeu Fabra, Barcelona, Spain

“Children’s identities and partial algorithmic identifications: A gap to bridge with 
human agency”

Giovanna Mascheroni, Catholic University of Milan, Italy

“Evaluating and monitoring AI applications in the classroom: Open Challenges”

Sole Pera, Bose State University, USA

”Children’s right to protection from economic exploitation in an AI-world"

Valerie Verdoodt, Ghent University, Belgium

6969

CHAPTER 9

9.1. Regulation and policies on artificial intelligence should consider the full 
range of children’s rights, especially when these conflict

Victoria Baines  
Bournemouth University, UK

Having worked in child online protection for many years, I have seen the very worst that malevolent 
adults  and  peers  can  do  to  children  and  young  people,  and  I  am  all  too  aware  of  the  long-term 
impact of child sexual exploitation and abuse on victims’ emotional and physical well-being. Not 
for nothing is the provision of specialist psycho-social support a requirement of both the Council 
of Europe Convention on Protection of Children against Sexual Exploitation and Sexual Abuse, also 
known as ‘the Lanzarote Convention’, and the We Protect Global Alliance Model National Response 
for preventing and tackling these crimes.⁷⁵

It stands to reason, then, that as citizens, caregivers, stakeholders and policymakers, we should do 
everything in our power to prevent children from experiencing child sexual exploitation and abuse in 
the first place. In addition to its primary goal of reducing the incidence of harm, effective prevention 
also alleviates the burden on law enforcement, the criminal justice sector, social care specialists and 
healthcare practitioners.

In recent years, prevention and detection of child sexual exploitation and abuse has been increasingly 
facilitated by technological innovation: many parents and caregivers now use apps that monitor and 
restrict  children’s  use  of  digital  communications;  law  enforcement  uses  big  data  analysis  to  find 
needles  in  investigative  haystacks;  hotlines  for  reports  of  child  sexual  abuse  material  use  image 
hashing  and  matching  to  minimise  vicarious  trauma  from  unnecessary  exposure  to  content  that 
has already been investigated; large technology companies use similar tools to conduct automated 
scanning for such material, which is then removed and reported to law enforcement. 

But protection from all forms of sexual abuse and exploitation enshrined in Article 34 of the United 
Nations  Convention  on  the  Rights  of  the  Child  is  not  the  only  right  to  which  children  and  young 
people are entitled. Indeed, among the articles of this binding instrument, children have the right to:

be free to express their thoughts and opinions and to access all kinds of information, as long as 
it is within the law (Article 13);

think and believe what they choose and also to practise their religion, as long as they are not 
stopping other people from enjoying their rights (Article 14);

privacy (Article 16)

the right to reliable information from a variety of sources, and governments should encourage 
the media to provide information that children can understand. Governments must help protect 
children from materials that could harm them (Article 17);

education (Article 28).

Consideration of these articles en masse reveals links but also potential conflicts between them. 
For instance, children are entitled to access “all kinds of information” but should be protected from 
“materials  that  could  harm  them.”  Moreover,  a  child’s  right  to  privacy  may  be  compromised  by 
measures  that  intrude  into  their  and  others’  private  lives  in  order  to  keep  them  safe.  Decisions 

NOTES
⁷⁵ https://www.coe.int/en/web/children/convention; https://www.weprotect.org/model-national-response/

7070

CHAPTER 9

concerning  the  correct  degree  of  compromise  not  infrequently  demand  careful  consideration  of 
trade-offs between different, sometimes conflicting rights. 

Despite best efforts, there can be unintended and unproductive consequences, a high-profile example 
being  the  extension  in  December  2020  of  the  scope  of  the  E-privacy  Directive  (2002/58/EC)  to 
“number-independent interpersonal communications services” such as voice chat, messaging and 
web-based mail. 

The Directive’s provisions for the confidentiality of communications and traffic data (Articles 5(1) 
and 6(1)) immediately prohibited social media platforms’ predominant method of scanning for child 
sexual abuse material. 

The  impact  on  detection  and  reporting  was  immediate,  the  US  National  Center  for  Missing  and 
Exploited  Children  (NCMEC)  observing  a  58%  reduction  in  reports  of  child  sexual  abuse  material 
from  the  largest  US  platforms  to  the  European  Union.⁷⁶  Considerable,  fast-time,  effort  by  child 
protection specialists and other stakeholders was required to ensure the passing of a temporary 
derogation from the problematic provisions.

Similar tensions apply to children’s use of digital technology. The proliferation of what has come to 
be known as ‘safety tech’ has been stimulated by significant government support in some countries.⁷⁷ 
Tools that conduct surveillance on children’s communications and activities, considered intrusive or 
‘stalkerware’ when used on adult subjects, are regularly marketed and deployed in the interest of 
child protection. As some developers are discovering, these solutions may contravene data protection 
legislation and prohibitions on unlawful interception of communications.⁷⁸

AI  accelerates  and  accentuates  this  tension:  as  Stanford  University’s  2021  AI  Index  Report  has 
observed,  “The  technologies  necessary  for  large-scale  surveillance  are  rapidly  maturing,  with 
techniques  for  image  classification,  face  recognition,  video  analysis,  and  voice  identification  all 
seeing significant progress in 2020”.⁷⁹ A growing number of safety tech products claim to make use 
of AI and/or Machine Learning to identify harmful content or risky behaviours. 

But what is harmful and risky does not always equate with what is illegal or criminal. Often these 
tools  target  pre-criminal  situations,  such  as  online  contact  from  an  unknown  adult,  or  searches 
outside  pre-determined  parameters.  The  aim  of  preventing  harm  and  criminal  activity  is  well-
placed.  But  it  may  well  come  at  a  price,  both  for  the  privacy  of  the  child  and  their  development 
of effective risk mitigation skills. Our understandable inclination to remove the risk of harm may 
have the unintended consequence of hindering children’s and young people’s resilience to unwanted 
experiences. 

From this perspective, and that of children’s rights to a private life and access to information, child 
protection stakeholders should perhaps ask ourselves, “Just because we can collect even more data 
from children and block even more content in order to protect them, does that mean we always 
should?”

NOTES
⁷⁶ https://www.missingkids.org/blog/2020/we-are-in-danger-of-losing-the-global-battle-for-child-safety

⁷⁷ https://www.gov.uk/government/publications/safer-technology-safer-users-the-uk-as-a-world-leader-in-safety-tech/safer-technology-
safer-users-the-uk-as-a-world-leader-in-safety-tech

⁷⁸ See, for example, the experience of UK company SafeToNet: https://www.theregister.com/2021/11/26/safetonet_message_scanning_
legal_warning/

⁷⁹ https://aiindex.stanford.edu/report/

7171

CHAPTER 9

9.2. AI education: Beyond mere transfer of knowledge

Tara Chklovski 
Technovation, USA

We know that sugar is bad for us, but that knowledge is still not enough to stop us from eating more 
than the daily recommended amounts (a meager 25 g!). In fact corporations thrive on adding more 
and more sugar in their products to increase sales. So knowledge itself is not enough to prevent bad 
decision making and action. 

When thinking about equipping, arming and empowering children to stay safe while using AI-based 
devices, our first reactions would be to institute protective regulations and in parallel to educate 
children on how the AI is working and its uses and dangers. Yes and yes! 

But in reality, this is a more complex issue and if we really want to make a difference, we need to 
go deeper. 

We need to not only teach children how these AI-based systems work, but to actually build and train 
their own AI models to tackle problems children care about. We all know that the best learning is 
when you actually apply, build, create, improve and innovate. If we know this, then we should not just 
stop at the knowledge transfer stage, but build systems and programs where children are learning 
about AI by applying, building and creating AI-based systems that make the world better for all of 
us. This is especially important when learning about something as intangible as AI systems, because 
it takes so much imagination and understanding of different human conditions and experiences, to 
be able to predict edge cases and when the AI could potentially harm others. 

How can we do this? 

Below is a 3-step plan to create an effective AI education program for children that not only educates 
them about how AI works, but also nurtures empathetic, kind innovators and inventors. 

* Step 1 - Learn to think about complex systems and how to train AI-based systems to tackle a 
complex, social problem, aligned with the UN sustainable development goals (ideal examples of 
really important, unsolved, complex, real-world problems that impact a huge number of people). 

* Step 2 - Learn to identify your own values, that of your family and community. This is a critical 
step in understanding what values are important to you and which ones are worth defending and 
protecting. 

* Step 3 - Learn to connect with different groups and communities, get their feedback and thoughts 

on the AI solution and build a map of edge cases. 

A hands-on, AI-education program that covers the three facets outlined above, will most likely last 
between 60-100 hours and is not for the faint of heart, but such a program will be transformational. 
In earlier societies, children did not have to understand complex concepts and systems to survive 
(although we have always had the exceptional dreamers and the explorers who went far afield!) 
But to survive and thrive, one didn’t need the sophisticated set of tools that we need now. Most 
importantly, our circle of influence has increased dramatically through technology. We can do good 
or  cause  damage  at  great  distances,  and  to  people  we  have  never  met.  These  are  much  harder 
concepts to understand, and to do so, will require time. 

But the rewards could not be greater. 

We need to invite children to be co-creators and co-designers of AI-based tools that can be used 
to tackle the big, complex problems we all face as a planet and as a people. And if we invest in 
them with depth and conviction, then we will not only have informed users today, but empathetic, 
innovative leaders tomorrow who truly care about making sure all of us, and our planet thrives.

7272

CHAPTER 9

9.3. Directions for the responsible design and use of AI by children and their 
communities: Examples in the field of Education

Davinia Hernández-Leo 
Universitat Pompeu Fabra, Barcelona

Our  digital  world  is  advancing  towards  an  increasingly  complex  ecosystem  driven  by  Artificial 
Intelligence technologies. Advances in the field of AI are often accomplished by actively exploring 
new techniques related to the collection and analysis of data to perform tasks normally requiring 
(and thus, potentially replacing) human intelligence as well as new tasks not achievable by humans. 
This scenario carries with it new risks that conflict with ethical principles, such as children rights.⁸⁰ 
Yet, a perspective focused on preventing children from using AI or from the effects of others using 
AI (e.g. their educators) is a strategy also to the detriment of their rights. 

On the one hand, applications of AI can bring benefits. For example, AI and data-driven technologies 
bring  opportunities  to  design  enhanced  educational  environments  for  children  to  learn.  Examples 
include systems that scaffold the development of learners’ self-regulation skills (Molenaar, 2021) or 
teacher-facing actionable learning analytics dashboards to orchestrate productive learning scenarios 
(Amarasinghe & Hernández-Leo, 2021), and many others (Tuomi, 2018). Indeed, children have the 
right to an education that enables them to fulfil their potential.⁸¹ 

On the other hand, children need to be prepared to live healthily in our real world. For example, social 
interactions,  entertainment  and  access  to  information  and  products  are  increasingly  occurring  in 
(social) media and web platforms, which are increasingly incorporating AI features. Indeed, children 
have the right to get ready for a life and a development in our real world. 

Education  and  risk  prevention  used  to  be  a  sufficiently  good  strategy  to  address  the  challenges 
identified in available digital technologies previous to AI. But risk prevention is not enough in a digital 
world with AI. What is rather needed is a responsible provision that considers ethics principles in 
the early conception of AI and data-driven technologies and its expected use. To actually care for 
the rights of the child in the design of AI technology, a policy and societal perspective should also 
consider  the  perspective  of  their  interplay  with  individual  children  and  their  communities.  In  the 
following, I suggest some methods towards this end.

Honest AI. Beyond trustworthy AI, AI applications should be designed in a way in which the intelligence 
of AI is continuously questioned. This design-thinking approach will often lead to hybrid AI-human 
systems that involve humans in the loop, actually giving them the control in a rights-based manner. 
For example, smart alerts visualised in a teacher-facing dashboard may explain predictions about 
potential learners’ failures or lack of engagement. The prediction is visualised with indicators about 
the trustworthiness of the prediction (e.g. related to fairness and level of transparency in the way 
the prediction is designed and presented) but recognises its ignorance about many elements in the 
specific context and their values. In such an example, the teacher will be able to react to the alert 
or not, depending on her own interpretation about how a derived action may benefit the children in 
their actual context. 

AI minimisation. In parallel to the concept of “data minimisation”, an approach to AI minimisation 
would  limit  the  use  of  AI  to  complete  tasks  that  are  really  necessary  to  accomplish  a  valuable 
purpose and without falling into a technological solutionism trap (Selbst et al., 2019). For example, 
in systems that scaffold the development of learners’ self-regulation skills, AI-based scaffolds are 

NOTES
⁸⁰ https://www.unicef.org/globalinsight/reports/policy-guidance-ai-children 

⁸¹ https://www.ohchr.org/en/professionalinterest/pages/crc.aspx 

7373

CHAPTER 9

helpful in early phases of skill development but they should fade out when the individual is ready to 
practice the meta-cognitive tasks without support. Yet, and despite there is already relevant research 
in the domain, even more research, and especially longitudinal and community-scale studies (see 
below), is needed to accumulate sufficient evidence about what actually are the “valuable purposes” 
of AI for children.

Agency  and  well-being  by  design.  AI  applications  and  systems  should  promote  children's  self-
fulfilment. More than inhibiting their decision making options, AI should be designed in a way that 
activates  their  agency  so  children  are  able  to  participate  in  decisions  about  matters  influencing 
themselves. Enacting agency leads to enhanced physiological well-being (e.g. sense of autonomy 
and capability) (Peters et al., 2018). Moreover, there are opportunities in the design of AI features to 
consider how to counter-effect the factors emerging from the use of digital technologies contributing 
to mental health conditions (e.g. FOMO in social media) and to focus on applications that support 
the  development  of  socio-emotional  skills  (e.g.  critical  thinking,  empathy,  stress  management) 
for  safeguarding  their  well-being.  Finally,  recommended  practises  for  assessing  the  impact  of  AI 
applications  on  children  well-being,  as  proposed  by  the  IEEE  for  human  well-being  (IEEE,  2020; 
sample application in Hakami & Hernández-Leo, 2021) would be also useful in the iterative design 
of AI for children and its impact in diverse societal dimensions.

Reduce  AI  gap  (or  AI  divide,  or  AI  exclusion).  AI  brings  a  new  perspective  to  the  already 
established  notion  of  “digital  gap”.  While  the  notion  of  digital  gap  has  been  more  connected  to 
how  unequal  access  to  ICT  leads  to  social  inequality,  the  notion  of  AI  gap  relates  more  to  how 
unequal access to AI literacy leads to higher levels of AI-derived risks for disadvantaged children 
when they (or others in their behalf) use AI-enabled systems. This effect is amplified if connected 
with the need of socio-emotional or well-being self-protection skills in the use of AI, as vulnerable 
populations are more likely to have suffered from early life well-being challenges and often have 
less opportunities to develop those skills. In a world where AI applications and systems are to be used 
also by disadvantaged populations, the design of those applications should consider this new notion 
of AI gap, both by involving those populations in the human-centred design of the technologies and 
by considering the gap in the design of how requirements like transparency should be implemented 
(i.e. the explanation required for an AI feature depends on the level of AI literary of the user). 

Embedded teachable moments. AI applications could embed educational features being activated 
when meaningful in the use of those applications. These features would offer “teachable moments” 
eventually playing two roles, favouring transparency and honest AI and facilitating skill development. 
For example, social media could embed educational features about the mechanisms and effects of 
filter bubbles (Hernández-Leo et al., 2021).

Community  approach.  Specific  contexts  may  lead  to  different  ethical  impacts  in  the  use  of  AI 
applications (e.g. vulnerable context mentioned above). This aspect has an effect on, for example, 
the way it is possible to answer questions about what should be the role of AI in formal education 
and informal learning. A community approach would be suitable to tackle this challenge. Through 
collective knowledge construction processes (Ley et al., 2020), where individuals create knowledge 
collaboratively as well as acquire attitudes and skills, communities can enact a collective agency 
influencing  decisions  regarding  purposes,  design  and  use  of  AI  in  their  context.  This  applies  to 
local  communities  involving  the  different  relevant  actors  (children,  parents,  schools,  associations, 
researchers, policymakers etc.) or to teacher communities (in a particular context or across contexts 
and geographies) willing to contribute to the sharing and co-design of strategies and the assessment 
of impacts.

7474

CHAPTER 9

9.4.  Children’s  identities  and  partial  algorithmic  identifications:  A  gap  to 
bridge with human agency

Giovanna Mascheroni 
Catholic University of Milan, Italy

When we think of children and AI we often think of some extraordinary, yet to come technology 
embodied  in  anthropomorphic  social  robots  able  to  interact  with  the  child  on  an  empathic  basis. 
Yet, children already encounter a variety of AI-based technologies in their everyday lives: from the 
recommendation systems of YouTube or SVOD platforms, to the conversational agents embedded 
in domestic smart speakers, the school platform’s algorithms, the facial recognition software that 
detects them in the photos shared by their parents, etc.  That these and other technologies are not 
recognised as AI suggests their profound integration into everyday life.

The invisibility and taken-for-grantedness of AI in children’s lives is a sign of both its commercial 
success and its social legitimation, which conceals the wider power structures on which AI depends. 
AI systems have been designed as infrastructures for massive data extraction, predictive analysis 
and monetisation within the business logic of surveillance capitalism (Zuboff, 2019).  The extension 
of  data-driven  processes  of  governance  from  the  market  to  ever  more  domains  of  social  life, 
including the intimacy of our homes, has been legitimised through discourses and imaginaries that 
present datafication and automation—the two key components of every AI system— as inevitable 
and desirable. It is the ideology of data, or “dataism” (Van Dijck, 2014), which sustains the pervasive 
colonisation and profitable annexation of everyday life through data extraction (Couldry & Mejias, 
2019).  To  put  it  simply,  children’s  everyday  lives—their  contexts,  practices,  and  emotions—are 
turned  into  profitable  resources  for  data  capitalism.  Therefore,  AI-based  technologies  are  never 
neutral: they are inscribed with a data-driven business model that rests upon the belief in a perfect 
conversion from life to data, and in data as natural traces and more objective knowledge. However, 
as I remind of in my latest book with my colleague Andra Siibak (Mascheroni & Siibak, 2021), every 
act of translation is always partial, a corruption in which something gets inevitably lost. 

As a consequence, a gap comes to exist between children’s lives and identities—nuanced, complex, 
messy—and  their  algorithmic  identifications—partial  and  inaccurate,  yet  absolute,  opaque  and 
therefore unchallenged. It is in such gap that the risks of AI for children’s rights are inscribed and 
enacted: decisions are made automatically by algorithms on children’s behalf, which are based on 
often inaccurate, partial data about them and in ways that are not accessible to them.

Researchers and policymakers now call for equity, inclusion and diversity in the design of AI systems 
(for  children).  However,  the  epistemic  gap  between  embodied  individuals  and  their  data  doubles, 
on one side, and the training of machine-learning based on historical datasets  on the other side 
complicate any attempt at making AI more inclusive, equitable and non-discriminatory. Algorithmic 
bias,  either  in  the  historical  data  used  to  train  machine-learning  or  in  the  very  abstraction  and 
classification of data or, probably, on both, turns the human errors that algorithms should replace 
into a systemic under- or over-representations of probabilities for certain populations, defined by 
their age or gender, ethnic group or education, income or residence, psychometrics, etc.  

This  generates  “allocative”  and  “representational  harms”  (Burkell  &  Bailey,  2018):  that  is,    the 
unequal  allocation  of  resources  legitimised  on  the  basis  of  presumably  “impartial”  algorithmic 
classifications,  and  the  influence  of  stereotyped  classifications  on  an  individual’s  representation, 
their understanding of the social world and, ultimately, their agency. There is raising concern that 
AI systems could inform unequal policies that systematically exclude certain categories of children 
from access to opportunities, with the effect of pre-determining their future. In order to ensure that 

NOTES
⁸² It is their lack of autonomy in learning which renders AI ultimately non intelligent, according to Crawford (2021).

7575

CHAPTER 9

data and algorithmic decision-making are re-purposed for the social good—i.e. to inform long-term 
evidence-based  preventive  programmes  to  ensure  that  vulnerable  children  have  equal  access  to 
resources—certainly we have to ask what data are collected, by whom, how and with what purposes.

This  is  what  regulation  around  children’s  data  protection  has  been  concerned  with.  And,  yet,  two 
more challenges arise in this regard. First, such legal frameworks often bring to the fore children’s 
rights to privacy and protection from commercial exploitation to the detriment of other children’s 
rights:  GDPR  Article  8,  for  example,  has  been  criticised  not  only  for  its  difficult  implementation, 
but also for potentially conflicting with children’s right to participation, freedom of expression and 
association, to information and media, education and play. 

Second, privacy itself is framed as an individual rather than social problem (Mascheroni, 2018). This 
is problematic, for the burden of privacy itself is placed mainly, if not solely, on parents. Furthermore, 
an individual approach on privacy fails to acknowledge that children’s data are often collected as 
part of their parents profiles, so that their own protection as vulnerable data subjects is hindered by 
the everyday practices of many. Moreover, in order to repurpose AI for a better future for children, we 
should not limit policy interventions to data protection regulation. We should also need to envision 
ways to let children have a voice in the automated decisions made for them by AI systems. That is, 
we need to bridge the gap between identities and algorithmic identifications with more, rather than 
less, human agency. 

9.5. Evaluating and monitoring AI applications in the classroom:  
Open Challenges
Sole Pera 
Boise State University, USA

The impact of Artificial Intelligence (AI) is far-reaching, as it now is seamlessly integrated behind 
the scenes into the daily lives of adults and children alike (Wong et al., 2020). Steered by the rights 
of  the  child,  there  is  a  setting  that  serves  as  the  perfect  ecosystem  to  reflect  on  the  trade-offs 
between the ample opportunities AI affords and the risks emerging from children's interactions with 
AI applications: the classroom. 

Why the classroom? AI applications—robots, recommender systems, conversational systems, and 
even search systems—are widely utilised by educators and children of all ages (Beelen et al., 2021; 
Allen et al. 2021; Ekstrand et al. 2020; Lovato et al. 2019; Pera et al. 2019; Murgia et al., 2019 
For example, consider the use of smart toys among preschoolers (Akdeniz & Özdinç, 2021) or the 
integration of AI-based assistants to aid skill development among high schoolers (du Boulay, 2016). 
The nature of the classroom provides a minimal set of requirements expected of AI applications. Still, 
the adoption of these applications is not without consequence, thus making this complex scenario 
the right one to identify how policy enforcing the rights of the child can drive design and regulate 
deployment. Where to start? I would argue in favour of using a particular application as a case study. 
A case study would enable identifying open challenges about design and deployment, along with 
exploring how to address these challenges. Moreover, it could facilitate organic policy development, 
assuring that AI applications empower and explicitly respond to their rights.

What are some of the known hurdles? Regardless of the application of choice—from a recommender 
system that offers reading materials to help promote literacy among the youth to search engines 
embedded in the classroom to support information discovery—the first step involves undertaking 
a holistic approach that regards the voices of all the stakeholders involved. It is critical to consider 
at the design stage, not as an afterthought, the needs, wants, and possibilities (from the user and 
system perspectives), if AI applications are to be safe and valuable for children while practical to 
deploy and adopt. In this setting, there are plenty of stakeholders to consider: children, of course, but 
also educators, parents, industry partners, researchers, and policymakers. Further, we must assess 
what developmental psychology tells us about cognitive development and the stages through which 
children progress that define the limits of what they can assimilate (Piaget, 1997) if aiming to treat 

7676

CHAPTER 9

AI applications as learning tools to expand knowledge (Hansen & Rieh, 2016). Most children, however, 
cannot develop the ability to use AI applications effectively by themselves without assistance. The 
technology itself could help via built-in scaffolding that explicitly considers developmental stages 
(Luckin & Cukurova 2019; Roschelle et al., 2000). Doing so is vital to ensure that for each child, 
AI applications are not only accessible but instructional as well. Lastly, taking theory to practice in 
this complex setting demands a broad range of experts actively informing design and deployment. 
Literacy and cognitive development experts, computer scientists (with expertise in natural language 
processing, machine learning, information retrieval, and child-computer interaction), in addition to 
experts in matters of privacy, law, and ethics, are only the starting point.

What are some of the primary considerations? Children have particular cognitive, social, physical, 
and emotional needs that make the information they seek, their experiences, their sense-making, 
and  their  skills  different  from  those  of  adults.  Paraphrasing  Bilal  (2010),  children  are  not  simply 
short users; they are unique users. As such, there are plenty of open questions that cannot merely be 
answered by turning to what we know about the design, deployment, and adoption of AI technology 
targeting adult users in varied domains. Among the more immediate situations to address, we find 
the trade-off between privacy and personalisation (a must for AI applications that support learning); 
the digital divide (as not all children have access to the same level of support when it comes to 
the proper use of AI applications for the classroom); the need to train the trainers (as the adults 
in  the  life  of  children  might  not  possess  the  know-how  to  impart  best  practices);  the  concept  of 
misinformation (as AI applications should offer content children can rely on), and, more importantly, 
the issue of interactions among engagement, distraction, and learning (as AI applications integrated 
in the classroom should not be a detriment to traditional teaching practices). 

What is next? Lessons learned from a case study can guide the deployment and integration into 
children's everyday lives of AI applications that are safe and attentive to their needs and expectations. 
AI applications that also abide by principles of fairness and, most of all, explicitly understand and 
respond to the fact that there is no such a thing as one-size-fits-all when it comes to children.

9.6. Children’s right to protection from economic exploitation in an AI-world

Valerie Verdoodt 
Law and Technology at Ghent University (BE)

Since the adoption of the United Nations Convention on the Rights of the Child in 1989, the living, 
playing and learning environment of children has changed dramatically. While AI-enabled toys and 
voice assistants have entered children’s homes, in schools, it’s AI-powered tutors, learning assistants 
and  personalised  educational  programmes  that  are  gaining  momentum.⁸³  Children’s  interactions 
and communications are also increasingly mediated by AI-enabled platforms and applications like 
social media, video-sharing- and interactive gaming platforms. These platforms rely on advanced 
machine  learning  to  deliver  content  and  personalise  –  or  in  their  own  words  “improve”⁸⁴    -  user 
experience and maximise engagement. 

The specific features of these technologies make them not only particularly appealing to children, 
but also potentially harmful . A particular challenge to children’s rights relates to the fact that many 
AI  technologies  children  engage  and  play  with  are  developed  by  private  companies  and  offered 
on a for-profit basis. This raises the concern that the inherently commercial and opaque nature of 
these  technologies  may  lead  to  an  increasingly  monetised  digital  playground  for  children,  often 
without them realising it (Van Der Hof et al. 2020; Verdoodt & Lievens, 2017; Verdoodt et al. 2016). 

NOTES
⁸³ UNICEF, The State of the World’s Children 2017: Children in a Digital World (UNICEF 2017) 58.

⁸⁴ For instance TikTok’s For You Feed, see https://newsroom.tiktok.com/en-us/how-tiktok-recommends-videos-for-you.

7777

CHAPTER 9

The  potential  misuse  of  AI  for  commercial  purposes  is  also  clearly  on  the  minds  of  children  and 
young people themselves (cfr. section 6.3). Therefore, one important question that deserves special 
attention from policymakers, researchers and other stakeholders is the following: Where should we 
draw the line between acceptable commercial practices and AI-enabled exploitation? 

This, of course, is not a simple question. Longitudinal studies on how the intrusion of commercial 
interests and monetisation techniques through AI-based applications and systems affects children’s 
long-term  well-being  and  development  are  still  missing.  Then  again,  even  if  there  is  no  (hard) 
evidence yet on the harmfulness of these practices to children’s well-being, scholars like LIEVENS 
have advocated a ‘better safe than sorry’-approach, based on the ‘precautionary principle’ (Lievens, 
2021). More specifically, she argues that “the principle compels society to act cautiously if there 
are certain – but not necessarily absolute – scientific indications of a potential danger and if not 
acting upon these indications could inflict harm” (Lievens, 2021). At present, exploitative practices 
in  the  digital  world  have  already  reached  the  point  where  they  are  specifically  designed  for  and 
are becoming ever more sophisticated in manipulating children into sharing more personal data or 
spending more money. 

Moreover,  when  (commercial)  persuasive  algorithms  dictate  children’s  decision-making  there  is  a 
danger that their autonomy, creativity and ultimately their development will be compromised . In 
response to this, the argument is often made that we need to empower children so that they can 
cope  with  the  commercial  pressures  that  these  new  technologies  bring.  We  see  this  for  instance 
with (proposed) legal and ethical frameworks (for AI) prescribing empowerment measures such as 
transparency and information rights as a means of protection. While such rights and measures are 
indeed crucial for children’s education and development, they are not a panacea. The lack of insight 
into how exactly AI-systems generate certain outputs makes it extremely difficult for children – or 
adults for that matter - to anticipate potential risks, harms or violations of their rights. 

In that respect, the responsibility for understanding how AI-systems operate, process data and make 
decisions, or assessing the fairness of outputs cannot be placed solely on the shoulders of children 
or  their  parents  (Lievens,  2021).  Instead,  a  precautionary  approach  requires  other  actors  such  as 
policymakers, businesses, and regulators to step up and ensure that children’s rights become a reality. 
When these actors make policies or (commercial) decisions on AI that may affect children, they must 
carefully assess their potential impact on all children’s rights - including their right to protection from 
economic exploitation - and take the best interests of the child as a primary consideration. During this 
process, children and young people must be actively listened to and their views must be effectively 
taken into account (Verdoodt 2021). Only when meaningful child participation becomes the norm in 
decision- and policy-making on technology will children’s rights be fully realised in an AI-world.

NOTES
⁸⁵ OECD, ‘Children in the Digital Environment - Revised Typology of Risks’ 5.

⁸⁶ (Van Der Hof et al. 2020; Verdoodt 2019). Thereby negatively impacting children’s rights to privacy (Article 16 UNCRC) and to protection 
from economic exploitation (Article 32 UNCRC).

⁸⁷ This is problematic from the perspective of Article 6 UNCRC. The Council of Europe’s Committee of Ministers, for instance, has warned 
that the “fine grained, sub-conscious and personalised levels of algorithmic persuasion may have significant effects on the cognitive 
autonomy of individuals and their rights to form opinions and take independent decisions” Council of Europe, Committee of Ministers, 
‘Declaration by the Committee of Ministers on the Manipulative Capabilities of Algorithmic Processes’.

⁸⁸ See for example: Proposal for a Regulation of the European Parliament and of the Council laying down harmonised rules on artificial 
intelligence (Artificial Intelligence Act) and amending certain Union legislative acts Com/2021/206 Final; AI HLEG, ‘Ethics Guidelines for 
Trustworthy AI’; Regulation (EU) 2016/679 (General Data Protection Regulation); UNICEF, ‘Policy Guidance on AI for Children’ (2021). See 
also Lievens (2020).

⁸⁹ Marietje Schaake, ‘The European Commission’s Artificial Intelligence Act’ 7, 2.

⁹⁰ In relation to this, various means should be explored, including (1) establishing legal restrictions for AI (e.g. Article 5 of the Proposal for 
the AI); (2) ensuring responsibilities for companies (and other actors) that develop and use AI technologies; (3) and having regulators with 
the necessary know-how and financial resources to monitor and handle complaints. See (Lievens, 2021; Verdoodt 2020).

⁹¹ Article 3 UNCRC and Article 24 CFEU.

7878

REFERENCES

N.B.  The  references  to  policy-related  reports  appear  as  footnotes  throughout  the  text. 
Below, we list the references to scientific works.

Akdeniz,  M.,  &  Özdinç,  F.  (2021).  Maya:  An  artificial  intelligence  based  smart  toy  for  pre-school 
children. International Journal of Child-Computer Interaction, 29, 100347.

Ali, M. R., Razavi, S. Z., Langevin, R., Al Mamun, A., Kane, B., Rawassizadeh, R., ... & Hoque, E. (2020, 
October).  A  virtual  conversational  agent  for  teens  with  autism  spectrum  disorder:  Experimental 
results and design lessons. In Proceedings of the 20th ACM International Conference on Intelligent 
Virtual Agents (pp. 1-8).

Allen, G., Yang, J., Pera, M. S., & Gadiraju, U. (2021). Using Conversational Artificial Intelligence to 
Support Children's Search in the Classroom. arXiv preprint arXiv:2112.00076.

Alsobhi,  A.  Y.,  &  Alyoubi,  K.  H.  (2019).  Adaptation  algorithms  for  selecting  personalised  learning 
experience based on learning style and dyslexia type. Data Technologies and Applications.

AL-Rossais, N., and D. Kudenko, D. (2018). Evaluating Stereotype and Non-Stereotype Recommender 
Systems. In Proceedings of KaRS workshop at RecSys.

Amarasinghe, I., Hernández-Leo, D., (2021). Synergies between humans and machines to support the 
orchestration of CSCL scripts at different scales, In Proceedings of the 14th International Conference 
on  Computer-Supported  Collaborative  Learning-CSCL  2021,  International  Society  of  the  Learning 
Sciences, Bochum, Germany (online), pp. 279-280.

Anuyah,  O.,  Milton  A.,  Green  M.,  and  Pera  M.  S.  (2019)  An  empirical  analysis  of  search  engines’ 
response to web search queries associated with the classroom setting. Aslib Journal of Information 
Management.

Bart, V. K., Sharavdorj, E., Bazarvaani, K., Munkhbat, T., Wenke, D., & Rieger, M. (2019). It was me: The 
use of sense of agency cues differs between cultures. Frontiers in psychology, 10, 650. 

Beelen, T., Velner, E.., Ordelman, R., Truong, K., Evers, V., & Huibers, T. (2021). Does your robot know? 
Enhancing  children's  information  retrieval  through  spoken  conversation  with  responsible  robots. 
arXiv preprint arXiv:2106.07931.

Bilal, D. (2010). The mediated information needs of children on the Autism Spectrum Disorder (ASD). 
In Proceedings of the 31st ACM SIGIR Workshop on Accessible Search Systems, Geneva, Switzerland 
(pp. 42-49). Geneva: ACM.

Borgers, N., De Leeuw, E., & Hox, J. (2000). Children as respondents in survey research: Cognitive 
development and response quality 1. Bulletin of Sociological Methodology/Bulletin de méthodologie 
sociologique, 66(1), 60-75. URL: https://doi.org/10.1177/075910630006600106

du  Boulay,  B.  (2016).  Artificial  intelligence  as  an  effective  classroom  assistant.  IEEE  Intelligent 
Systems, 31(6), 76-81.

Brščić, D., Kidokoro, H., Suehiro, Y., & Kanda, T. (2015). Escaping from children's abuse of social robots. 
In Proceedings of the tenth annual acm/ieee international conference on human-robot interaction 
(pp. 59-66).

Burkell,  J.,  &  Bailey,  J.  (2018).  Unlawful  distinctions?  Canadian  human  rights  law  and  algorithmic 
bias. In 2016/2018 Canadian yearbook for human rights (pp. 217–230). Human Rights Research and 
Education Centre. https://cdp-hrc.uottawa.ca/sites/cdp-hrc.uottawa.ca/files/uottawa-cyhr1618-v11.pdf

7979

Catania, F., Crovari, P., Beccaluva, E., De Luca, G., Colombo, E., Bombaci, N., & Garzotto, F. (2021). 
Boris:  a  Spoken  Conversational  Agent  for  Music  Production  for  People  with  Motor  Disabilities.  In 
CHItaly 2021: 14th Biannual Conference of the Italian SIGCHI Chapter (pp. 1-5).

Charisi, V., Gomez, E., Mier, G., Merino, L., & Gomez, R. (2020a). Child-Robot collaborative problem-
solving and the importance of child's voluntary interaction: a developmental perspective. Frontiers in 
Robotics and AI, 7, 15.

Charisi, V., Malinverni, L., Schaper, M. M., & Rubegni, E. (2020b). Creating opportunities for children's 
critical reflections on AI, robotics and other intelligent technologies. In Proceedings of the 2020 ACM 
Interaction Design and Children Conference: Extended Abstracts (pp. 89-95).

Charisi, V., Merino, L., Escobar, M., Caballero, F., Gomez, R., & Gómez, E. (2021a). The Effects of Robot 
Cognitive Reliability and Social Positioning on Child-Robot Team Dynamics. In 2021 IEEE International 
Conference on Robotics and Automation (ICRA) (pp. 9439-9445). IEEE.

Charisi, V., Imai, T., Rinta, T., Nakhayenze, J. M., & Gomez, R. (2021b). Exploring the Concept of Fairness 
in  Everyday,  Imaginary  and  Robot  Scenarios:  A  Cross-Cultural  Study  With  Children  in  Japan  and 
Uganda. In Interaction Design and Children (pp. 532-536).

Charisi, V., Sabanovic, S., Gasser U., Gomez, R. (2021c). Social Robots and Children’s Fundamental 
Rights: A Dynamic Four-Component Framework for Research, Development, and Deployment. Paper 
presented at the WeRobot 2021, Miami, USA. https://werobot2021.com/wp-content/uploads/2021/08/
Charisi-et_al_Social_robots_and_Children_VER3.pdf

Chaudron,  S.,  Beutel,  M.,  Cernikova,  M.,  Navarette,  V.  D.,  Dreier,  M.,  &  Fletcher-Watson,  B.  (2015). 
Young children (0-8) and digital technology. A qualitative exploratory study across seven countries. 
Joint Research Centre. European Commission. doi:10. 2760/294383. 

Chaudron, S., Geneiatakis, D., Kounelis, I., & Di Gioia, R. (2019). Testing internet of toys designs to 
improve privacy and security. In The Internet of Toys (pp. 223-239). Palgrave Macmillan, Cham.

Cheng, Y., Yen, K., Chen, Y., Chen, S., & Hiniker, A. (2018). Why doesn't it work? voice-driven interfaces 
and young children's communication repair strategies. In Proceedings of the 17th ACM Conference 
on Interaction Design and Children (pp. 337-348).

Couldry, N., & Mejias, U. A. (2019). The costs of connection: How data is colonizing human life and 
appropriating it for capitalism. Stanford University Press.

Crawford, K. (2021). The Atlas of AI. Yale University Press.

Dautenhahn,  K.  (2007).  Socially  intelligent  robots:  dimensions  of  human–robot  interaction. 
Philosophical transactions of the royal society B: Biological sciences, 362(1480), 679-704.

Davison, D. P., Wijnen, F. M., Charisi, V., van der Meij, J., Evers, V., & Reidsma, D. (2020, March). Working 
with a social robot in school: a long-term real-world unsupervised deployment. In Proceedings of the 
2020 ACM/IEEE International Conference on Human-Robot Interaction (pp. 63-72). 

Dias, P., Brito, R., Ribbens, W., Daniela, L., Rubene, Z., Dreier, M., Gemo, M., di Gionia, R., & Chaudron, S. 
(2016). The role of parents in the engagement of young children with digital technologies: Exploring 
tensions between rights of access and protection, from ‘Gatekeepers’ to ‘Scaffolders’. Global Studies 
of Childhood, 6(4), 414-427. https://doi.org/10.1177/2043610616676024

Downs, B., French, T., Landau Wright, K., Pera, M. S., Kennington, C., & Fails, J. A. (2019). Children and 
search tools: Evaluation remains unclear. In KidRec Workshop co-located with ACM IDC 2019.

8080

Druga, S., Williams, R., Breazeal, C., & Resnick, M. (2017). "Hey Google is it ok if I eat you?" Initial 
explorations in child-agent interaction. In Proceedings of the 2017 conference on interaction design 
and children (pp. 595-600).

Ekstrand, M. (2017). Challenges in evaluating recommendations for children. In International 
Workshop on Children & Recommender Systems. Available at: 
https://drive. google. com/file/d/0B8S_mL8gIjZwVzZSLVZKN3Q4RG8/view. 

Ekstrand, M. D., Wright, K. L., & Pera, M. S. (2020). Enhancing classroom instruction with online news. 
Aslib Journal of Information Management.

Fails, J. A., Pera, M. S., Garzotto, F., & Gelsomini, M. (2017). KidRec: Children & recommender 
systems: Workshop co-located with ACM conference on recommender systems (recsys 2017).  
In Proceedings of the Eleventh ACM Conference on Recommender Systems (pp. 376-377). 
https://doi.org/10.1145/3109859.3109956. 

Fitter, N. T., Chowdhury, Y., Cha, E., Takayama, L., & Matarić, M. J. (2018). Evaluating the effects of 
personalized appearance on telepresence robots for education. In Companion of the 2018 ACM/IEEE 
International Conference on Human-Robot Interaction (pp. 109-110). 

Fitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017). Delivering cognitive behavior therapy to young adults 
with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a 
randomized controlled trial. JMIR mental health, 4(2), e7785.

Fraser, J., Papaioannou, I., & Lemon, O. (2018). Spoken conversational ai in video games: Emotional 
dialogue  management  increases  user  engagement.  In  Proceedings  of  the  18th  International 
Conference on Intelligent Virtual Agents (pp. 179-184).

Garg, R., & Sengupta, S. (2020). He is just like me: a study of the long-term use of smart speakers 
by parents and children. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous 
Technologies, 4(1), 1-24. 

Ghiglino, D., Chevalier, P., Floris, F., Priolo, T., & Wykowska, A. (2021). Follow the white robot: Efficacy 
of robot-assistive training for children with autism spectrum disorder. Research in Autism Spectrum 
Disorders, 86, 101822.

Gómez, E., Charisi, V., & Chaudron, S. (2021). Evaluating recommender systems with and for children: 
towards a multi-perspective framework, co-located with the 15th ACM Conference on Recommender 
Systems, Amsterdam, The Netherlands, 2021. URL: http://ceur-ws.org/Vol-2955/paper2.pdf

Gómez,  E.,  Charisi,  V.,  Tolan,  S.,  Miron,  M.,  Martinez  Plumed,  F.  and  Escobar  Planas,  M.,  HUMAINT: 
Understanding  the  impact  of  Artificial  Intelligence  on  human  behaviour,  Amran,  G.  editor(s), 
Publications Office of the European Union, Luxembourg, 2021, ISBN 978-92-76-28212-9 (online), 
978-92-76-28211-2 (print), doi:10.2760/23970 (online), 10.2760/043359 (print), JRC122667.

Gordon, G., Spaulding, S., Westlund, J. K., Lee, J. J., Plummer, L., Martinez, M., ... & Breazeal, C. (2016, 
March).  Affective  personalisation  of  a  social  robot  tutor  for  children’s  second  language  skills.  In 
Proceedings of the AAAI conference on artificial intelligence (Vol. 30, No. 1). 

Goyal, N., Wice, M., Aladro, A., Kallberg-Shroff, M., & Miller, J. G. (2019). Culture and the development 
of views of agency: Perspectives from storybooks, parents, and children. Developmental psychology, 
55(5), 1096.

Guo,  W.,  Gao,  H.,  Shi,  J.,  Long,  B.,  Zhang,  L.,  Chen,  B.  C.,  &  Agarwal,  D.  (2019,  July).  Deep  natural 
language processing for search and recommender systems. In Proceedings of the 25th ACM SIGKDD 
International Conference on Knowledge Discovery & Data Mining (pp. 3199-3200).

8181

Hakami,  E.,  &  Hernandez-Leo,  D.  (2021).  Investigating  the  Well-being  Impacts  of  Educational 
Technologies  Supported  by  Learning  Analytics:  An  application  of  the  initial  phase  of  IEEE  P7010 
recommended  practice  to  a  set  of  cases.  In  LAK21:  11th  International  Learning  Analytics  and 
Knowledge Conference, online,  pp. 269-279. 

Hamlin, J. K., & Baron, A. S. (2014). Agency attribution in infancy: Evidence for a negativity bias. PloS 
one, 9(5), e96112.

Hansen, P., & Rieh, S. Y. (2016). Recent advances on searching as learning: An introduction to the 
special issue.

Hernández-Leo, D., Theophilou, E., Lobo, R., Sánchez-Reina, R., & Ognibene, D. (2021). Narrative scripts 
embedded  in  social  media  towards  empowering  digital  and  self-protection  skills.  In  ECTEL2021: 
European Conference on Technology Enhanced Learning, pp. 394-398.

Izci, B., Jones, I., Özdemir, T. B., Alktebi, L., & Bakir, E. (2019). Youtube & young children: research, 
concerns and new directions. Crianças, famílias e tecnologias. Que desafios? Que caminhos?, 81-92.

Johal, W. (2020). Research trends in social robots for learning. Current Robotics Reports, 1-9.

T.  Johns  (2021).  “Parents  of  children  called  Alexa  challenge  Amazon”.  BBC.  https://www.bbc.com/
news/technology-57680173

Kahn  Jr,  P.  H.,  Kanda,  T.,  Ishiguro,  H.,  Freier,  N.  G.,  Severson,  R.  L.,  Gill,  B.  T.,  ...  &  Shen,  S.  (2012). 
“Robovie,  you'll  have  to  go  into  the  closet  now”:  Children's  social  and  moral  relationships  with  a 
humanoid robot. Developmental psychology, 48(2), 303.

Kanda, T., Hirano, T., Eaton, D., & Ishiguro, H. (2004). Interactive robots as social partners and peer 
tutors for children: A field trial. Human–Computer Interaction, 19(1-2), 61-84.

Keizer, S., Guhe, M., Cuayáhuitl, H., Efstathiou, I., Engelbrecht, K. P., Dobre, M., ... & Lemon, O. (2017). 
Evaluating persuasion strategies and deep reinforcement learning methods for negotiation dialogue 
agents. ACL.

Kennedy J, Lemaignan S, Montassier C, Lavalade P, Irfan B, Papadopoulos F, Senft E, Belpaeme T. 
(2017) Child Speech Recognition in Human-Robot Interaction: Evaluations and Recommendations, 
http://doi.org/10.1145/2909824.3020229

Kraus, M., Kraus, J., Baumann, M., & Minker, W. (2018). Effects of gender stereotypes on trust and 
likability in spoken human-robot interaction. In Proceedings of the eleventh international conference 
on language resources and evaluation (LREC 2018).

Landoni,  M.,  Murgia,  E.,  Huibers,  T.,  &  Pera,  M.  S.  (2020).  You've  Got  a  Friend  in  Me:  Children  and 
Search Agents. In Adjunct Publication of the 28th ACM Conference on User Modeling, Adaptation and 
Personalisation (pp. 89-94). 

Lavechin M., Bousbib R., Bredin H., Dupoux E., Cristia A. (2020), An open-source voice type classifier 
for child-centered daylong recordings, arXiv:2005.12656

Law, E., Baghaei Ravari, P., Chhibber, N., Kulic, D., Lin, S., Pantasdo, K. D., ... & Dillen, N. (2020). Curiosity 
Notebook: A Platform for Learning by Teaching Conversational Agents. In Extended Abstracts of the 
2020 CHI Conference on Human Factors in Computing Systems (pp. 1-9). 

Levin E. (2011) Child Development. In: Goldstein S., Naglieri J.A. (eds) Encyclopedia of Child Behavior 
and Development. Springer, Boston, MA. https://doi.org/10.1007/978-0-387-79061-9_523

8282

Ley,  T.,  Maier,  R.,  Thalmann,  S.,  Waizenegger,  L.,  Pata,  K.,  &  Ruiz-Calleja,  A.  (2020).  A  Knowledge 
Appropriation Model to connect scaffolded learning and knowledge maturation in workplace learning 
settings. Vocations and Learning, 13, 91−112.

Lievens,  E.  (2020).  The  rights  of  the  child  in  the  digital  environment:  from  empowerment  to  de-
responsibilisation. In Freedom, security and privacy: the future of childhood in the digital world (pp. 
152-157). 5rights Foundation. 

Lievens,  E.  (2021).  Growing  Up  with  Digital  Technologies:  How  the  Precautionary  Principle  Might 
Contribute  to  Addressing  Potential  Serious  Harm  to  Children’s  Rights.  Nordic  Journal  of  Human 
Rights, 39(2), 128-145. 

Liu, R., Gailhofer, P., Gensch, C., Köhler, A., Wolff, F. (2019). Impacts of the digital transformation on 
the environment and sustainability, Issue Paper under Task 3 from the “Service contract on future 
EU environment policy”, Öko-Institut, Institute for Applied Ecology.

Livingstone, Sonia (2014) Children's digital rights: a priority. Intermedia, 42 (4/5). pp. 20-24. ISSN 
0309-118X 
http://eprints.lse.ac.uk/60727/1/__lse.ac.uk_storage_LIBRARY_Secondary_libfile_
shared_repository_Content_Livingstone%2C%20S_Childrens%20digital%20rights_Livingstone_
Childrens%20digital%20rights_2015.pdf

Lovato,  S.  B.,  Piper,  A.  M.,  &  Wartella,  E.  A.  (2019).  Hey  Google,  do  unicorns  exist?  Conversational 
agents as a path to answers to children's questions. In Proceedings of the 18th ACM International 
Conference on Interaction Design and Children (pp. 301-313).

Luckin, R., & Cukurova, M. (2019). Designing educational technologies in the age of AI: A learning 
sciences‐driven approach. British Journal of Educational Technology, 50(6), 2824-2838.

Mascheroni, G. (2018). Researching datafied children as data citizens. Journal of Children and Media, 
12(4), 517–523. https://doi.org/10.1080/17482798.2018.1521677

Mascheroni, G., & Siibak, A. (2021). Datafied Childhoods: Data Practices and Imaginaries in Children’s 
Lives. Peter Lang.

McTear, M. (2020). Conversational AI: dialogue systems, conversational agents, and chatbots. 
Synthesis Lectures on Human Language Technologies, 13(3), 1-251. 
https://doi.org/10.2200/S01060ED1V01Y202010HLT048

Milton,  A.,  Murgia,  E.,  Landoni,  M.,  Huibers,  T.,  &  Pera,  M.  S.  (2019).  Here,  there,  and  everywhere: 
Building a scaffolding for children’s learning through recommendations. IMPactRS2019 Workshop – 
co-located with ACM RecSys 2019. CEUR Workshop Proceedings, 2462, 2019. 

Milton, A. and Pera, M. S. (2020). The Horror: Evaluating Information Retrieval Systems for Kids”. In 
Proceedings of the 4th International and Interdisciplinary Perspectives on Children & Recommender 
and Information Retrieval Systems (KidRec 2020), co-located with ACM IDC 2020.

Molenaar, I. (2021). Towards hybrid human-AI regulation: Supporting young learners' self-regulated 
learning. EARLI 2021: Book of Abstracts, 256.

Mondada, F., Bonnet, E., Davrajh, S., Johal, W., & Stopforth, R. (2016). R2T2: Robotics to integrate 
educational efforts in South Africa and Europe. International Journal of Advanced Robotic Systems, 
13(5), 1729881416658165.

Murgia, E., Landoni, M., Huibers, T., Fails, J. A., & Pera, M. S. (2019). The seven layers of complexity of 
recommender systems for children in educational contexts. ComplexRec Wokrshop – Co-located with 
ACM RecSys 2019. CEUR Workshop Proceedings, 2449, 5-9, 2019.

8383

Narayanan,  S.,  &  Potamianos,  A.  (2002).  Creating  conversational  interfaces  for  children.  IEEE 
Transactions on Speech and Audio Processing, 10(2), 65-78., http://doi.org/10.1109/89.985544

Nasihati  Gilani,  S.,  Traum,  D.,  Merla,  A.,  Hee,  E.,  Walker,  Z.,  Manini,  B.,  ...  &  Petitto,  L.  A.  (2018). 
Multimodal dialogue management for multiparty interaction with infants. In Proceedings of the 20th 
ACM International Conference on Multimodal Interaction (pp. 5-13). 

Neumann,  M.  M.  (2020).  Social  robots  and  young  children’s  early  language  and  literacy  learning. 
Early Childhood Education Journal, 48(2), 157-170.

Nomura, T., Kanda, T., Kidokoro, H., Suehiro, Y., & Yamada, S. (2016). Why do children abuse robots?. 
Interaction Studies, 17(3), 347-369.

Olszewska,  J.  I.  (2020).  IEEE  Recommended  Practice  for  Assessing  the  Impact  of  Autonomous 
and  Intelligent  Systems  on  Human  Well-Being:  IEEE  Standard  7010-2020.,  DOI:  10.1109/
IEEESTD.2020.9084219

Papert, S. (1993). The children's machine: Rethinking school in the age of the computer. BasicBooks, 
10 East 53rd St., New York, NY 10022-5299.

Pera, M. S., & Ng, Y. K. (2014). Automating readers' advisory to make book recommendations for k-12 
readers. In Proceedings of the 8th ACM Conference on Recommender Systems (pp. 9-16).

Pera, M. S., Murgia, E., Landoni, M., & Huibers, T. (2019). With a little help from my friends: Use of 
recommendations  at  school.  In  2019  ACM  Conference  on  Recommender  Systems  Late-breaking 
Results, ACM RecSys LBR 2019 (pp. 61-65). CEUR.

Peters, D., Calvo, R., & Richard., R. (2018). Designing for Motivation, Engagement and Wellbeing in 
Digital Experience. Frontiers in Psychology. 9. DOI: 10.3389/fpsyg.2018.00797.

Piaget  J.  (1997).  Development  and  learning.  In:  Gauvain  M.  &  Cole  G.  M.  (eds.)  Readings  on  the 
development of children. Second Edition. W. H. Freeman, New York: 19–28. Available at: http://cepa.
info/3043.

Picton, I. (2014). The Impact of eBooks on the Reading Motivation and Reading Skills of Children and 
Young People: A Rapid Literature Review. National Literacy Trust.

Pradhan, A., Mehta, K., & Findlater, L. (2018). "Accessibility Came by Accident" Use of Voice-Controlled 
Intelligent Personal Assistants by People with Disabilities. In Proceedings of the 2018 CHI Conference 
on human factors in computing systems (pp. 1-13).

Radesky, J. S., Weeks, H. M., Ball, R., Schaller, A., Yeo, S., Durnez, J., ... & Barr, R. (2020).  
Young children’s use of smartphones and tablets. Pediatrics, 146(1).  
https://pediatrics.aappublications.org/content/146/1/e20193518. doi:10. 1542/peds.2019-3518.

Raj,  A.,  Milton,  A.,  &  Ekstrand,  M.  D.  (2021).  Pink  for  Princesses,  Blue  for  Superheroes:  The  Need 
to  Examine Gender Stereotypes in Kid's  Products in  Search  and  Recommendations. arXiv preprint 
arXiv:2105.09296. URL: https://arxiv.org/abs/2105.09296. arXiv:2105.09296.

Ricci,  F.,  Rokach,  L.,  &  Shapira,  B.  (2011).  Introduction  to  recommender  systems  handbook.  In 
Recommender systems handbook (pp. 1-35). Springer, Boston, MA. doi:10.1007/978-0-387-85820-3.

Roschelle, J. M., Pea, R. D., Hoadley, C. M., Gordin, D. N., & Means, B. M. (2000). Changing how and 
what children learn in school with computer-based technologies. The future of children, 76-101.

8484

Røyneland, K. (2020). It knows how to not understand us!. A study on what the concept of 
robustness entails in design of conversational agents for preschool children.  
https://www. duo. uio. no/handle/10852/69059. Accessed, 21.

Schedl, M., & Bauer, C. (2017). Online Music Listening Culture of Kids and Adolescents., KidRec 2017 
https://arxiv.org/abs/1912.11564

Sciuto, A., Saini, A., Forlizzi, J., & Hong, J. I. (2018). " Hey Alexa, What's Up?" A Mixed-Methods Studies 
of In-Home Conversational Agent Usage. In Proceedings of the 2018 Designing Interactive Systems 
Conference (pp. 857-868).http://doi.org/10.1145/3196709.3196772

Selbst,  A.  D.,  Boyd,  D.,  Friedler,  S.  A.,  Venkatasubramanian,  S.,  &  Vertesi,  J.  (2019).  Fairness  and 
abstraction in sociotechnical systems. In Proceedings of the Conference on fairness, accountability, 
and transparency, pp. 59-68.

Sezgin, E., Noritz, G., Elek, A., Conkol, K., Rust, S., Bailey, M., ... & Huang, Y. (2020). Capturing at-home 
health and care information for children with medical complexity using voice interactive technologies: 
multi-stakeholder viewpoint. Journal of medical Internet research, 22(2), e14202.

Shiomi, M., Nakata, A., Kanbara, M., & Hagita, N. (2017). A hug from a robot encourages prosocial 
behavior. In 2017 26th IEEE international symposium on robot and human interactive communication 
(RO-MAN) (pp. 418-423). IEEE.

Song-Nichols, K., & Young, A. (2020). Gendered Robots Can Change Children's Gender Stereotyping. 
In CogSci.

Søraa, R. A., Nyvoll, P. S., Grønvik, K. B., & Serrano, J. A. (2021). Children’s perceptions of social robots: 
a study of the robots Pepper, AV1 and Tessa at Norwegian research fairs. AI & society, 36, 205-216. 

Spezzano,  F.  (2021).  Using  Service-Learning  in  Graduate  Curriculum  to  Address  Teenagers' 
Vulnerability to Web Misinformation. In Proceedings of the 26th ACM Conference on Innovation and 
Technology in Computer Science Education V. 2 (pp. 637-638)

Straten,  C.  L.  V.,  Peter,  J.,  Kühne,  R.,  &  Barco,  A.  (2020).  Transparency  about  a  robot's  lack  of 
human psychological capacities: effects on child-robot perception and relationship formation. ACM 
Transactions on Human-Robot Interaction (THRI), 9(2), 1-22.

Strubell, E., Ganesh, A., & McCallum, A. (2019). Energy and policy considerations for deep learning in 
NLP. in 57th Annual Meeting of the Association for Computational Linguistics (ACL). Florence, Italy. 
July 2019   https://arxiv.org/abs/1906.02243v1

Tsiakas, K., Barakova, E., Khan, J. V., & Markopoulos, P. (2020). BrainHood: towards an explainable 
recommendation system for self-regulated cognitive training in children. In Proceedings of the 13th 
ACM International Conference on PErvasive Technologies Related to Assistive Environments (pp. 1-6).

Tuomi, I., The Impact of Artificial Intelligence on Learning, Teaching, and Education, Cabrera Giraldez, 
M., Vuorikari, R. and Punie, Y. editor(s), EUR 29442 EN, Publications Office of the European Union, 
Luxembourg, 2018, ISBN 978-92-79-97257-7, DOI: 10.2760/12297, JRC113226

Ueno, M., & Miyazawa, Y. (2017). IRT-based adaptive hints to scaffold learning in programming. IEEE 
Transactions on Learning Technologies, 11(4), 415-428. 

UNICEF (2021) AI and Child’s Rights: Case study: The Haru Robot https://www.unicef.org/globalinsight/
media/2206/file

8585

Van Der Hof, S., Lievens, E., Milkaite, I., Verdoodt, V., Hannema, T., & Liefaard, T. (2020). The child’s 
right to protection against economic exploitation in the digital world. The International Journal of 
Children's Rights, 28(4), 833-859.

Van Dijck, J. (2014). Datafication, dataism and dataveillance: Big Data between scientific paradigm 
and ideology. Surveillance & society, 12(2), 197-208. https://doi.org/10.24908/ss.v12i2.4776

Vaughn, L. M., & Jacquez, F. (2020). Participatory Research Methods – Choice Points in the Research 
Process. Journal of Participatory Research Methods, 1(1). https://doi.org/10.35844/001c.13244

Verdoodt, V., Clifford, D., & Lievens, E. (2016). Toying with children's emotions, the new game in town? 
The legality of advergames in the EU. Computer Law & Security Review, 32(4), 599-614.

Verdoodt, V., & Lievens, E. (2017). Targeting children with personalised advertising: how to reconcile 
the  (best)  interests  of  children  and  advertisers.  In  Data  Protection  and  Privacy  Under  Pressure: 
Transatlantic tensions, EU surveillance, and big data (pp. 313-341). Maklu.

Verdoodt, V. (2019). The role of children’s rights in regulating digital advertising. The International 
Journal of Children's Rights, 27(3), 455-481. 

Verdoodt,  V.  (2020).  Children's  rights  and  commercial  communication  in  the  digital  era  (Vol.  10). 
Intersentia. 

Verdoodt, V. (2021). Children’s rights in the digital environment: Moving from theory to practice.

Vincent-Lancrin, S. (2021). Frontiers of smart education technology: Opportunities and challenges. 
OECD Digital Education Outlook 2021 Pushing the Frontiers with Artificial Intelligence, Blockchain 
and Robots.

Vinuesa, R., Azizpour, H., Leite, I., Balaam, M., Dignum, V., Domisch, S., ... & Fuso Nerini, F. (2020). The 
role of artificial intelligence in achieving the Sustainable Development Goals. Nature communications, 
11(1), 1-10. https://doi.org/10.1038/s41467-019-14108-y

Wang, C., Wang, K., Bian, A., Islam, R., Keya, K. N., Foulde, J., & Pan, S. (2021). Bias: Friend or Foe? 
User  Acceptance  of  Gender  Stereotypes  in  Automated  Career  Recommendations.  UMBC  Student 
Collection. arXiv preprint arXiv:2106.07112., 2021.  

Wang, Z., Huang, J., & Fiammetta, C. (2021). Analysis of Gender Stereotypes for the Design of Service 
Robots: Case Study on the Chinese Catering Market. In Designing Interactive Systems Conference 
2021 (pp. 1336-1344).

Wong, G. K., Ma, X., Dillenbourg, P., & Huan, J. (2020). Broadening artificial intelligence education in 
K-12: where to start?. ACM Inroads, 11(1), 20-29.

Xu, Y., & Warschauer, M. (2020). "Elinor Is Talking to Me on the Screen!" Integrating Conversational 
Agents into Children's Television Programming. In Extended Abstracts of the 2020 CHI Conference 
on Human Factors in Computing Systems (pp. 1-8).

Zaman, B. (2020). Designing Technologies with and for Youth: Traps of Privacy by Design. Media and 
Communication, 8 (4), 229-238. doi: 10.17645/mac.v8i4.3261 

Zhang, L., Weitlauf, A. S., Amat, A. Z., Swanson, A., Warren, Z. E., & Sarkar, N. (2020). Assessing social 
communication and collaboration in autism spectrum disorder using intelligent collaborative virtual 
environments. Journal of autism and developmental disorders, 50(1), 199-211.

Zuboff, S. (2019). The age of surveillance capitalism: The fight for a human future at the new frontier 
of power. Profile Books.

8686

List of figures

Figure 1: Structure of the report  

Figure 2: Block diagram of conversational agents (Adapted from Mansfield et al., 2019).  

  5

29

Figure 3: Young participants’ preconceptions and perceptions of AI technology applications (blue team) 

38

Figure 4: Young participants’ preconceptions and perceptions of AI technology applications (green team)  39

Figure 5: Young workshop participants’ questions on future directions 

Figure 6: Brofenbrenner’s Ecological System 

Figure 7: Questions received from policymakers (N=28)  

Figure 8: Questions emerged through the discussions with experts (N=22) 

Figure 9: Topics posed by children and youth (N=35) 

43

48

61

61

61

List of tables

Table 1: Key actions proposed by the “EU strategy on the rights of the child” regarding the support 
of children’s rights in the digital and information society [adopted by the EU Strategy on the Rights 
 19 
of the Child] 

Table  2:  Emerging  opportunities  in  relation  to  children’s  rights,  in  the  context  of  recommender 
systems, conversational agents and social robots for children, as found in the relevant literature 

 25

Table  3:  Emerging  risks  in  relation  to  children’s  rights  in  the  context  of  recommender  systems, 
 25
conversational agents and social robots for children, as found in the relevant literature   

Table 4: Indicative questions generated during the Experts’ workshops 

 57

8787

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
GETTING IN TOUCH WITH THE EU 

In person 

All over the European Union there are hundreds of Europe Direct information centres. You can find the 
address of the centre nearest you at: https://europa.eu/european-union/contact_en 

On the phone or by email 

Europe Direct is a service that answers your questions about the European Union. You can contact this 
service: 

- by freephone: 00 800 6 7 8 9 10 11 (certain operators may charge for these calls), 

- at the following standard number: +32 22999696, or 

- by electronic mail via: https://europa.eu/european-union/contact_en 

FINDING INFORMATION ABOUT THE EU 

Online 

Information about the European Union in all the official languages of the EU is available on the Europa 
website at: https://europa.eu/european-union/index_en 

EU publications 
You can download or order free and priced EU publications from EU Bookshop at: 
https://publications.europa.eu/en/publications. Multiple copies of free publications may be obtained by 
contacting Europe Direct or your local information centre (see https://europa.eu/european-
union/contact_en). 

 
 
 
 
 
K
J
-
N
A
-
3
1
0
4
8
-
E
N
-
N

doi:10.2760/012329  

ISBN 978-92-76-51837-2