A Systematic Literature Review of Research on
AI Literacy from 2020 to 2024
First Author1,2*, Second Author2,3† and Third Author1,2†
1*Department, Organization, Street, City, 100190, State, Country.
2Department, Organization, Street, City, 10587, State, Country.
3Department, Organization, Street, City, 610101, State, Country.
*Corresponding author(s). E-mail(s): <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>;
Contributing authors: <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>; <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>;
†These authors contributed equally to this work.
Abstract
To be added
Keywords: to be added
1 Introduction
• What is the goal of this paper?
With the development of society, technology is also rapidly changing. In our modern
society, the use of AI is becoming increasingly widespread. Understanding what is
AI and how to use AI correctly is the central theme of this article. Focusing on this
theme, we will gradually understand the components of AI Literacy through the
study of the following questions: 1. What is AI? 2. What can AI do? 3. How does
AI work? 4. How should AI be used? 5. How do people perceive AI? Through the
study and analysis of the above five points, readers will gain a basic understanding
of the concept and uses of AI Literacy.
• What do we know about the topic from related work?
• What is the motivation for doing this work? Why is this work important?
• What are we gonna read in the next sections?
This article is AI literacy, with a focus on conducting a systematic literature review
1
of research on AI literacy from 2020 to 2024. This article explores the definition and
importance of AI literacy, including the abilities to understand, use, evaluate, and
create AI applications. Our research team referenced a set of AI literacy competencies proposed by Long, D. and Magerko, B. (2020) [1], analyzing 50 papers related
to AI literacy. We used a coding sheet to record AI literacy information from each
paper and applied Cohen’s kappa method to assess coding consistency. Additionally, this study analyzed the demographics of each paper, including the number of
authors, their nationalities, and research fields. In conclusion, this article compares
the research findings with existing frameworks, identifies current gaps in AI literacy research, and suggests future research directions, offering specific guidance for
future research and educational practice.
2 Related Work
2.1 AI Literacy
- AI Literacy Definitions & Importance
- What is AI Literacy? Competencies and Design Considerations (Long & Magerko)
The presence of artificial intelligence (AI) in modern life necessitates a new form
of literacy: AI Literacy. The widespread integration of AI into everyday applications underscores the need for broader public understanding. AI Literacy builds upon
established concepts like digital literacy and computational thinking, empowering individuals to become informed participants in the increasingly AI-driven world and it
could be crucial for reasons like:
• Informed Decision-Making: Helps individuals make informed choices about AI
applications and their impact on society.
• Effective Use of AI Tools: Equips users with the skills to utilize AI technologies
effectively in various contexts.
• Ethical Engagement: Encourages consideration of ethical implications, such as bias,
privacy, and transparency, promoting responsible use of AI.
• Career Competitiveness: Prepares individuals for the evolving job market, where AI
skills are increasingly in demand.
Conceptualizations of AI Literacy are continuously evolving. Long and Magerko’s article takes a step towards a) defining what competencies users need in order to effectively
interact with and critically evaluate AI and b) explaining how to design learnercentered
AI technologies that foster increased user understanding of AI.The authors conducted
an exploratory review of 150 interdisciplinary literature from before 2000 to 2019.
They did not conduct a traditional systematic literature review. Their methods were
instead more closely aligned with an approach called scoping studies, which aim to
map rapidly the key concepts underpinning a research area and the main sources and
types of evidence available.[2]
According to Long and Magerko,AI Literacy comprises three main competencies:
2
• Foundational Knowledge: Understanding basic AI concepts such as machine learning, neural networks, natural language processing, and computer vision.Recognizing
the potential and limitations of AI, identifying scenarios where AI can be effectively
applied and where human intervention is necessary.
• Practical Skills: Developing proficiency in using AI tools and applications, from
everyday AI-powered devices to specialized software.
Learning to interpret AI outputs and integrating AI into problem-solving and decisionmaking processes.Gaining hands-on experience through projects and simulations
involving building, training, and deploying AI
• Ethical Understanding: Cultivating awareness of the ethical implications of AI, such
as issues of bias, privacy, and transparency.
Promoting critical thinking about the societal impacts of AI, including job displacement, surveillance, and data security.Advocating for responsible AI usage, adhering to
ethical guidelines and best practices in AI development and deployment. Beyond these
core components, research is exploring the applications of AI Literacy within specific
environments. Studies on AI Literacy in digital work spaces, for example, highlight the
importance of understanding AI functionalities to leverage them for problem-solving.
Additionally, continuous learning about evolving AI trends and the ability to collaborate effectively with AI systems are crucial skills for navigating the modern workplace.
Long and Magerko emphasize some design considerations for developing effective AI
literacy programs suggesting that AI literacy education should be inclusive and accessible to diverse audiences, tailoring content to different age groups and educational
backgrounds. The authors advocate for incorporating hands-on, experiential learning
activities that engage learners actively and make abstract AI concepts tangible. Additionally, they highlight the importance of embedding ethical considerations throughout
the curriculum, ensuring that learners not only understand AI technology but also its
broader societal impacts.
2.1.1 Ulf-Daniel Ehlers et al. (2023)
Another AI Literacy framework called AI COMP focuses on answering the question
”which competences are important for being, becoming and remaining able to act
successfully in a professional and life world permeated by AI?”.
Unlike the framework proposed by Long & Magerko, the AI COMP framework is
based on an action competence model, which tries to define the conditions for people
to successfully perform actions using AI technologies in professional or private settings. The framework takes into account not only knowledge and skills, which form
the ability to act, but also inner attitudes and values, which lead to the willingness
to act. This framework is divided into three different levels. The first level is about
3 competence areas, which are (1) Work and create with AI and for AI, (2) Personal
Skills for AI-related areas of action and (3) Shaping the social environment with and
for AI. To these three competence areas, 12 competences have been assigned.
3
• Work and Create with AI: Activity and Implementation Competence, SystemDesign Competence, Creative Problem-Solving Competence, Critical Design Competence
• Personal Skills: Decision-Making Competence, Self-Efficacy, Critical Thinking,
Active control Competence, Self-determination
• Social Environment: Ethical Competence, Cooperation Competence, Communication Competence.
Finally the last level contains exemplary knowledge, skills and value attitudes for
every competence field. It is possible to conclude that, because of the different focuses
of each AI literacy framework, the frameworks differ in their modeling strategy and
listed competences, which shows the complexity of defining a generalized AI literacy
framework and generalized AI literacy competences.
2.1.2 Kelly Mills et al. (2024)
The framework from ”AI Literacy: A Framework to Understand, Evaluate, and Use
Emerging Technology” [3] connects Digital Promise [4] (global nonprofit working to
expand opportunity for each learner) to the safe and effective use of AI, which requires
informed users. It urges school leaders and policymakers to promote AI literacy, ensuring everyone can use emerging technologies responsibly. Also, the study is divided into
two parts. The first part describes an AI literacy framework that outlines practices
for understanding, evaluating, and using AI. The second part focuses on strategies for
educational leaders to promote this literacy among their specific audiences.
The way users engage in understanding and evaluating AI systems and tools are,
according to the paper, underpinned by core values. They emphasize the importance
of centering human judgment and centering justice in any application of AI. To achieve
this, clear communication is essential. This can be accomplished by breaking down
complex topics like AI literacy into practical steps that can help leaders define and
implement AI literacy programs in their specific contexts.
Some of the initiatives that educators have been discussing and implementing in
response to emerging technologies for decades are computational thinking, data literacy, digital citizenship, and media literacy. This approach enables school leaders to
expand on existing initiatives and learning pathways to include AI literacy or introduce additional learning objectives alongside AI literacy initiatives. Also, leaders can
have the foundation for building the capacity to teach AI literacy and will be better
equipped to apply these practices to AI tools if they have frequent opportunities to
develop them early on and across various grades and subject areas.
Then, three sections were presented: ”Understanding AI” provides an overview of
the information that leaders should adapt to their specific contexts, including a brief
overview of the history of AI, how it works, and its applications. In the ”Evaluating
AI” section, it is emphasized that ”we must constantly consider the benefits and/or
costs of AI to individuals, society, and the environment.” This involves understanding
and applying core values, which ”are overarching practices that support learners to
safely and effectively evaluate AI tools.” Two essential core values are named: Centering Human Judgment and Centering Justice. Leaders have a responsibility to link
4
these core values to the specific applications of AI systems and tools in their unique
contexts. By doing so, they provide ”additional context and explanation of how and
why to evaluate AI systems and tools,” ensuring that AI literacy development is both
comprehensive and contextually relevant. Finally, ”Using AI” describes three types of
its use: interacting, creating, and problem solving.
Also, it was given five strategies to promote AI literacy in K–12 education: provide
guidance for adoption and evaluation, integrate AI literacy across grades and subjectarea learning, facilitate ongoing, just-in-time professional learning, design powerful
learning experiences and promote awareness and agency.
By proposing these strategies, the paper helps people, in a very complete way, to make
informed decisions about if and how to use AI in different contexts. Some of these
strategies provides guidance for adoption and evaluation, integrating AI literacy across
grades and subject-area learning, facilitating professional learning, designing powerful
learning experiences, and promoting awareness and agency.
2.1.3 Other AI Literacy Frameworks and Long & Magerko
Theme Competences Long & Magerko AI COMP Digital Promise
What is AI?
Recognizing AI x x
Understanding Intelligence x x
Interdisciplinary x x x
General vs. Narrow x x
What can AI do? AI’s Strengths & Weaknesses x x x
Imagine Future AI x x
How does AI work?
Representations x
Decision Making x x
ML Steps x x
Human Role in AI x x
Data Literacy x x
Learning from Data x x
Critically Interpreting Data x x x
Action & Reaction x
Sensors x
How should AI be used? Ethics x x x
How Do People Perceive AI? Programmability x x x
Work and Create with AI for AI
Activity and implementation x
System-Design x
Creative problem-solving x
Personal Skills for AI-related areas
Self-efficacy x
Active control x
Self-determination x
Social environment with and for AI Cooperation x
Communication x
Table 1: Framework Comparison
The majority of competencies addressed to the second framework were addressed
to Long & Magerko, however, since there is a 5-year gap between these two papers, the
most recent one has details more adapted to current reality. For instance, in Long &
5
Theme Competences Long & Magerko AI COMP Digital Promise
What is AI?
Recognizing AI x x
Understanding Intelligence x x
Interdisciplinary x x x
General vs. Narrow x x
What can AI do? AI’s Strengths & Weaknesses x x x
Imagine Future AI x x
How does AI work?
Representations x
Decision Making x x
ML Steps x x
Human Role in AI x x
Data Literacy x x
Learning from Data x x
Critically Interpreting Data x x x
Action & Reaction x
Sensors x
How should AI be used? Ethics x x x
How Do People Perceive AI? Programmability x x x
Work and Create with AI for AI
Activity and Implementation x
System-Design x
Creative Problem-Solving x
Personal Skills for AI-related Areas
Self-Efficacy x
Active Control x
Self-Determination x
Social Environment with and for AI Cooperation x
Communication x
Table 2: Framework Comparison
Magerko, the theme ”How AI should be used” is addressed by only one competency:
”Ethics”, while Kelly Mills et al. [3] describes three types of use of AI: interacting,
creating and problem solving with AI. These three are also addressing different practices like ”Data Privacy & Security”, ”Information & Mis/Disinformation” and more,
one of today’s most talked-about subjects when it comes to technology itself.
Since AI comp focuses more on competences that make a person be able to act
successfully in a professional, private and social life environment shaped by AI, a lot
of competences are unique to AI COMP. For example Self-efficiacy, the ability to
solve AI Related issues by yourself or active control competence, which is about being
capable of personalizing Applications for ones own personal or private development or
self-determination competence, which is about being capable of using AI Applications
autonomously and confidently.
The three AI Literacy frameworks show both similarities and distinctiveness. Their
difference can arise from cultural contexts, intended audience, and research methodology. AI COMP- developed by the KI-Campus Hub Baden-Wuerttemberg focuses on
competencies relevant to professionals. Derived from empirical research, it identifies 12
areas of competencies. This model emphasizes practical skills and future relevance. AI
Literacy, proposed by Digital Promise, centers on AI Literacy for learning. Designed for
K-12 education, it defines AI Literacy as a set of competencies enabling critical evaluation, effective collaboration, and practical use of AI. In contrast, Long and Magerko’s
framework provides a more concrete set of competencies for AI. Through the analysis
of interdisciplinary literature, their learner-centered design approach addresses different core competencies and design considerations. The three frameworks were developed
at different points in time, spanning a five-year gap. It is important to recognize the
dynamic AI landscape that changed in this timespan that can also influence their
divergence.
6
2.2 Systematic Literature Reviews on AI Literacy
2.2.1 Casal-Otero et al. (2023)
A comprehensive systematic literature review by Casal-Otero et al. (2023) [5] offers
valuable insights into the integration of AI literacy in K-12 education. This review
analyzed 179 documents sourced from Scopus, categorizing the research into two
main groups: learning experiences and theoretical perspectives.
Key Findings:
Learning Experiences: Most of the AI literacy efforts focused on supporting students to acquire technical, conceptual and applied skills through multiple approaches:
games, robotics, and interactive visualizations. One major gap found was the absence
of assessments to measure pupils’ understanding of AI concepts consequent to learning
experiences.
Theoretical Perspectives:
The efforts in this category focus on setting up frameworks, guidelines, and curricula for AI literacy in schools. The main contributors in these efforts are from the
U.S. and China.
Recommendations for Integration of AI Literacy:
First and foremost, AI literacy should be built into the K-12 curriculum via an
interdisciplinary and competency-based approach. This would improve learning in core
subjects by integrating techniques from AI.
The second is the role of the teacher in AI literacy. There hence exists a need to
prioritize their training so far as the employment of new methodologies and tools for
teaching AI is concerned.
Third, the ethical implications of AI with respect to its societal impacts and
influence on critical thinking need consideration.
This means that a clear competency framework for AI literacy should be devised. It
would structure and assess learning outcomes so that students gain deep understanding
in AI.
Implications for Future Research:
The Casal-Otero et al. review reveals that in most cases, there had not been a series
of studies focused on easily evaluating the effectiveness of AI literacy programs. This
recommends future studies, giving comprehensive evaluations for AI literacy programs
since new ones have just sprung into action.
Furthermore, important are the development and validation of relevant AI literacy
competency frameworks.
Moreover, a diversity of subjects integrated with AI education is a way of promoting
interdisciplinary learning, which is very prestigious in itself.
This literature review raises awareness about the current state of affairs regarding
AI literacy in K-12 education by pointing out existing gaps and proposing concrete
steps toward closing them. It has been shown that a clearly specified competence
framework, specially targeted teacher education, and attention to ethical concerns
are some of the basic requirements underpinning the execution of AI literacy efforts.
7
2.2.2 Link¨oping University: In search of artificial intelligence (AI)
literacy in teacher education: A scoping review
The article [6] reviews the emerging topic of artificial intelligence literacy in teacher
education, highlighting its importance for teachers’ professional knowledge and practical use in class. ”The search strategy included papers and proceedings from 2000 to
2023 related to AI literacy and TE (teacher education) as well as the intersection of AI
and teaching.” Sperling et al. [6] By analyzing 34 papers, the review uses Aristotelian
concepts: episteme (theoretical knowledge), techne (practical knowledge), and phronesis (professional judgment), to examine how AI literacy is conceptualized in relation
to teacher education (TE). Key findings include:
- Emerging Research Topic: AI literacy is globally recognized in education but is
underrepresented in TE contexts.
- Diverse Topics and Methodologies: The literature spans various topics and
methods, influenced by computer science and exploratory teaching.
- Epistemic, Practical, and Ethical Knowledge: The research addresses theoretical and practical aspects but lacks a comprehensive focus on teachers’ professional
knowledge.
- Ethics: Ethical considerations are primarily technical, focusing on understanding
AI technologies.
- Practical Knowledge: Teachers’ practical knowledge often involves using digital
resources to teach AI or integrating AI EdTech into classrooms.
The review identifies significant gaps, particularly in practical and ethical knowledge, and calls for more comprehensive AI literacy education in TE. It aims to inform
future research and improve AI literacy in teacher education.
The article also discusses the concept of AI literacy, defining it as a set of competencies that allows individuals to critically evaluate AI technologies, communicate
and collaborate with AI, and utilize AI as a tool in various settings such as online,
at home, and at work. This definition by Long and Magerko is widely accepted and
is closely related to the idea of AI readiness, which refers to the understanding and
application of AI technologies in education and other areas. It furthermore notes that
while the concept of ”AI literacy” is frequently mentioned in many papers (23 in total),
definitions and methods to operationalize and measure it vary widely. Only a few
papers delve into what AI literacy actually entails, relying on conceptual frameworks
from Long and Magerko, Ng, and Luckin et al. These frameworks are preliminary and
not specifically tailored for different teacher and student groups but are used to map
teacher competencies and attitudes in specific contexts.
AI literacy is often discussed alongside other literacies like media, digital, and data
literacy, and is sometimes used interchangeably with terms such as AI readiness, AI
education, and AI thinking. The article highlights that only one study, an intervention with 34 secondary school Computer Science teachers by Vazhail et al. in 2021,
specifically explored AI literacy in teacher education (TE).
8
2.2.3 Artificial intelligence literacy in higher and adult education:
A scoping literature review
- The article evaluates the current state of AI literacy in higher and adult education through a scoping literature review, aiming to identify thematic foci and recent
research trends concerning AI literacy among non-experts due to the growing relevance of artificial intelligence in daily and professional life. The authors searched ten
research databases, initially finding 902 records and narrowing them down to 30 studies
using predefined eligibility criteria. The review focused on thematic areas, definitions
of AI literacy, target academic disciplines, and courses aimed at fostering AI literacy.
Findings revealed that research on AI literacy in higher and adult education is still in
its early stages and requires further refinement, with a lack of consensus on AI literacy definitions, varied thematic focuses, and differing educational approaches across
disciplines. The authors provided recommendations for future research and practical
implementation.
2.2.4 Omaima Almatrafi et al. (2024)
This systematic review synthesizes insights from 47 articles published between 2019
and 2023, aiming to capture new trends and initiatives in AI literacy. The review identifies three primary themes within the literature:
Conceptualizing AI Literacy:
Defining and understanding what AI literacy entails.
Promoting AI Literacy Efforts:
Strategies and initiatives to enhance AI literacy across various demographics. Developing AI Literacy Assessment Instruments: Tools and methodologies for
measuring AI literacy effectively.
Target Populations:
The articles addressed a diverse range of populations:
Pre-K to K-12 Students: Early education efforts to introduce AI concepts.
Higher Education Students: University-level programs and courses focusing on AI.
Adults in the Workforce: Training and development programs aimed at professionals
to improve their AI skills. Core constructs A thorough content analysis revealed six
key constructs essential to AI literacy:
Recognize: Identifying AI technologies and understanding their presence.
Know and Understand: Gaining knowledge about AI concepts and functions.
Use and Apply: Practical application of AI tools and technologies.
Evaluate: Assessing the effectiveness and implications of AI technologies.
Create: Developing new AI solutions and technologies.
Navigate Ethically: Understanding and addressing ethical considerations related to
AI.
The findings highlight the importance of enhancing AI literacy for both formal education and workforce development. By equipping individuals with the necessary skills to
leverage AI technologies, the review emphasizes the potential for driving innovation
9
and empowering individuals in an AI-driven world. The insights provided are valuable for educators, policymakers, and organizations aiming to improve AI literacy and
prepare individuals for future challenges and opportunities.
2.2.5 Davy Ng et al. (2021)
The review by Davy Ng et al. (2021) analyzes 30 peer-reviewed articles published
between 2016 and 2021, focusing on the definition, teaching methods, and assessment
strategies of AI literacy. AI literacy is defined as the ability to understand, use, evaluate, and create AI applications. The study uses Bloom’s Taxonomy to map AI literacy
to cognitive domains, finding that most studies focus on foundational understanding
and application, with less emphasis on higher-order analysis and creation. This indicates that AI literacy, as an emerging field, requires further research and educational
strategies to fully enhance students’ abilities in AI.
The study highlights the use of age-appropriate tools and innovative teaching methods, such as gamification, inquiry-based learning, and project-based learning, to help
students understand and engage with AI concepts. Quantitative assessments (including pre- and post-tests and surveys) measure knowledge acquisition and attitudes,
while qualitative methods (such as observations, interviews, and portfolio assessments)
evaluate students’ actual understanding and application of AI.
The article emphasizes that ethical considerations are an essential part of AI
literacy, focusing on understanding bias, accountability, transparency, and social
responsibility. The review underscores the importance of teaching students to use AI
ethically and consider its broader social impacts. Future AI literacy education should
be human-centered, ensuring that AI technology development and application take
into account diverse stakeholders and social inclusivity. It suggests integrating AI
literacy across all educational levels and disciplines, providing ongoing professional
development for teachers, and addressing resource limitations and standardized curriculum needs. The review calls for empirical research to evaluate the effectiveness of
AI literacy programs and develop robust assessment tools, emphasizing the importance of a comprehensive educational framework and ethical considerations to prepare
students for an AI-driven future.
In summary, this systematic literature review provides a comprehensive theoretical framework for the definition, teaching methods, and ethical issues of AI literacy
and offers guidance for future research and educational practice.
3 Methodology
3.1 Method
Initially, we were assigned the article ”What is AI Literacy? Competencies and Design
Considerations” [1]. For the first task was to find a definition for AI Literacy, report
the definition for its origin and how this definition was established. In the following week, we received an Excel spreadsheet called the coding sheet with 21 columns,
which containing the paper IDs (from number 1 to 50, with each number representing
10
a paper); the paper links (clicking on the links will redirect you to the corresponding
paper pages for reading); 17 columns of AI Literacy descriptions (used for the evaluators to pick out the corresponding AI Literacy information after reading the paper;
the number of authors; the nationality of the authors; the author’s field of study; the
main research methods used in the paper.
Afterwards we collected 50 AI-related papers from the ACM Digital
Library(https://dl.acm.org), which were divided into 10 groups ´a 5 papers. We then
coded and analysed these papers according to the choises in the provided spreadsheet
for four weeks. The first week we read and analyse 25 papers of 50, coding the spreadsheet. In the second week, we analyzed the coding results from the previous week,
exchanged opinions on-site, and generated a new unified coding result. In the third
week, we read and analyzed the remaining 25 papers, and each person did the coding
based on their own ideas. In the fourth week, we discussed and analyzed again to unify
the opinions on the results from the previous week. Then we collected all the coding
into a single spreadsheet, and used the Cohen’s kappa method to determine the final
results, assessing whether the papers contain the content of the 17 AI Literacy items.
Coding Process: A total of 12 people were involved into the task of reading papers
and coding them. These 12 people were all from Duisburg Essen University and were
students of the ”AI Literacy Seminar”. The 12 people were divided into several groups
of two. Each group was assigned to a different paper group, and two people in the same
group read five papers from the same paper group. Based on the content of the paper,
the two people selected the AI Literacy in Excel that matched the paper and labeled
it with a ”1” under the competency number. If the competency does not match the
content of the paper, then mark ”0”. After they read each paper, they also need to
fill in the areas like how many authors were in this paper, all these areas listed after
column Competency 17 in Excel.
Tabel 2 is a briefly presents the contents in our spreadsheet, introducing different
contents divided into three sections.
Paper Information AI Literacy Paper Demographics
Paper ID Competency 1 (Recognizing AI) Number of authors
Paper Link Competency 2 (Understanding Intelligence) Affiliation country of first author
Competency 3 (Interdisciplinarity) Affiliation area of first author
Competency 4 ( General vs. Narrow) Main methodology
Competency 5 (AI’s Strengths & Weaknesses) (interview, questionnaire,literature review,
Competency 6 (Imagine Future AI) logfile analysis, other (specify))
Competency 7 (Representations)
Competency 8 (Decision-Making)
Competency 9 (ML Steps)
Competency 10 (Human Role in AI)
Competency 11 (Data Literacy)
Competency 12 (Learning from Data)
Competency 13 (Critically Interpreting Data)
Competency 14 (Action & Reaction)
Competency 15 (Sensors)
Competency 16 (Ethics)
Competency 17 (Programmability))
Table 3: Excel spreadsheet for coding
11
3.2 Dataset
The dataset consists of the completed coding sheets from the reviewed research papers,
rated by two raters based on a semi-structured method. Each coding sheet contains
20 columns with information on various AI literacy competencies and other relevant
details. The raters indicated whether each competency was present in the papers they
reviewed. After the analysis we decided to remove paper Id 2 and 46 from our Dataset
because they were not a good fit for the resech question.
3.3 Analysis
After each pair of students individually coded their assigned competencies with a focus
mostly on the abstract and conclusion of the paper, we held an in-person meeting
to discuss our findings with our partner and resolve any disagreements in the coding
results. To ensure a minimum level of agreement between the the raters, we calculated
Cohen’s kappa between the ratings of (student 1 & student 2) and the discussion
results. Because not only AI Literacy but also leaning something about the literature
review process was the focus of our seminar and we did not yet reach the point were
we talked about different kappas, we used this approach to compare the raters. If
the kappa value exceeded 0.35, we retained the second-round coding; if not, a third
analysis was conducted by the assigned writer for each competency.
For metrics that could not be assigned binary values (such as the number of
authors, the country of the first author, and the publication year), the review process
was skipped because these were objective statistics not subject to different interpretations. For Competency 16: Ethics, the papers were further divided into subgroups
to analyze which parts of the ethics framework were studied.
4 Results
We used the AI Literacy Taxonomy proposed by [1].
4.1 Demographics
4.1.1 How many authors per paper?
The statistical analysis shows the following results:
• Average number of authors per paper: 3.73
• Standard deviation of the number of authors per paper: 2.05
Paper IDs per Category of Number of Authors
The provided figure and table offer an insightful overview of the distribution of the
number of authors per paper. Figure 1 displays a bar chart showing how papers are
distributed according to the number of authors involved. The chart indicates that the
most common number of authors per paper is three, followed by two and four authors,
while papers with one, five, six, and seven authors are less common. Papers with eight
and nine authors are relatively rare.
12
Fig. 1: Distribution of the Number of Authors per Paper
Number of Authors Paper IDs
1 13, 32, 38, 39, 41, 46
2 16, 24, 27, 28, 31, 33, 42, 43
3 1, 9, 12, 14, 17, 19, 22, 23, 26, 29, 40, 44, 47, 48, 49
4 3, 5, 20, 36
5 7, 10, 15, 21, 25, 35
6 18, 30, 45
7 6, 8, 11, 37, 50
8 4
9 34
Table 4: Paper IDs per Category of Number of Authors
Table 6 complements this visual data by listing specific paper IDs categorized
by the number of authors. This detailed breakdown shows the exact distribution of
papers for each category, making it easier to identify and reference specific papers
within each author group. For instance, papers with three authors include IDs such
as 1, 9, 12, 14, etc., while papers with a single author are identified by IDs like 13, 32,
38, etc. This combination of graphical and tabular data helps to understand the collaborative patterns in the research, highlighting a tendency towards moderate-sized
author groups rather than single authors or very large teams.
4.1.2 How many papers per country
We looked at the affiliation country of the first author of each collected paper and
13
listed them in Table 5. We also listed the amount of papers released by each country
in Fig. 2. Most of the papers we took into account for in our literacy review come from
the USA, with a total of 28 papers published. Out of the 48 papers published, 7 papers
were published in Germany. The rest of the countries published one or two papers.
Country Paper IDs
Australia 32
Canada 37
Chile 31
China 30, 38
Cyprus 12
Denmark 10, 18
Germany 16, 17, 24, 40, 43, 47, 50
Italy 14, 19
Malta 7
Spain 29
UK 34
USA 1, 3, 4, 5, 6, 8, 9, 11, 13, 15, 20,
21, 22, 23, 25, 26, 27, 28, 33, 35
36, 39, 41, 42, 44, 45, 48, 49
Table 5: Paper IDs per Country
4.1.3 How many papers per year?
4.2 What is AI?
Competency 1. [Recognizing AI...]
In the context of AI Literacy, the ability to distinguish between technological artifacts
that utilize Artificial Intelligence (AI) and those that do not is crucial. Out of the 48
assessed papers, 6 (12,5%) addressed this competency (inter-rater agreement: Cohen’s
kappa 0.78 - indicating substantial agreement) (Paper ID 3 [7], 4 [8], 12 [9], 24 [10], 41
[11], 45 [12]). For instance, in the context of fostering public AI literacy, paper 3 [13]
explores a hands-on exhibit called ”Knowledge Net” where people build network tiles
- a type of AI knowledge that represents relationships between objects. Learners were
Publication Year Paper IDs
2020 22, 27, 30, 34, 44, 48
2021 4, 8, 10, 17, 19, 24, 26, 28, 33, 40, 42, 49
2022 1, 5, 11, 20, 21, 23, 29, 36, 41, 50
2023 3, 6, 7, 9, 12, 13, 15, 18, 31, 35, 38, 39, 43, 46
2024 14, 16, 25, 32, 37, 45, 47
Table 6: Paper IDs per Category of years
14
Fig. 2: Distribution of the Number of Papers per Country
able to consider differences between human intelligence and the network. Paper 4 [8]
introduces an innovative AI Book Club model for professional development aimed at
building a foundational knowledge of AI for teachers, all while fostering a supportive
community. One key weekly meeting topic directly addressed ”What is AI - Examples
of AI”, aiming to equip educators with core AI literacy skills. Similarly, paper 12 [9]
highlights the importance of recognizing AI in everyday life. It describes an eight-week
course, ”AI in everyday life”, aimed at educating the public about AI technologies
and their implications. Paper 24 [10] focuses on developing a framework of core AI
competencies for non-computer science students. Through semi-structured interviews,
researchers highlighted the relevance of students acquiring basic AI competencies, such
as defining what AI is. These competencies cover a range of perspectives, from basic
AI understanding to more complex knowledge.
Competency 2. [Understanding Intelligence]
This skill involves analyzing and discussing the features that make an entity intelligent, distinguishing between human, animal, and machine intelligence. Definitions
vary, with some emphasizing foresight and adaptability, or highlighting traits like
15
Fig. 3: Distribution of the Number of Papers per Country
communication and learning. It also includes understanding empirical versus rational
approaches to intelligence. In total, 10 out of 48 papers (20%) addressed this competency (kappa: 0.35 / Fair Agreement) (Paper IDs: 7[14], 9[15], 12[9], 15[16], 29[17],
32[18], 33[19], 36[20], 38[21], 45[12]). For example paper 48[22] seeks to identify gaps
in the current explainable AI (XAI) algorithmic work and the questions users might
have to these intelligent systems. Or paper 9[15], which studies, how museums can
implement exhibits and demonstrations of AI with focus on many different Competencies from the field of AI literacy. Including a exhibit which was designed with the
goal to teach the visitors a better understanding of different types of intelligence.
Competency 3. [Interdisciplinarity] Understand that AI is used in many different
areas and is built with various technologies. Recognize different technologies used to
develop AI. This competency was studied in 22 out of 50 papers (PaperIDs 1[23],
4[8], 7[14], 8[24], 9[15], 10[25], 15[16], 16[26],19[27], 21[28], 22[29], 24[10], 29[17], 31[30],
33[19], 34[31], 36[20], 37[32], 39[33], 44[34],45[12], 47[35]) This corresponds to 44%. An
inter-rater agreement of 3.9 was calculated for this statistic, which is a fair agreement.
This means the evaluators generally had consistent opinions when determining whether
a paper studied this competency. In simpler terms, this fair agreement suggests that
different evaluators mostly agreed on which papers covered the topic, indicating that
their assessments were reasonably reliable but not perfect. This helps ensure that the
reported statistic—that 44% of the papers discussed AI development technologies—is
trustworthy. For example, paper 8 mentions AI being used in smartphones, social
media, and school surveillance[24]. This corresponds to the Competency ”interdisciplinarity” because it mentions different areas in which AI is being used. Similarly,
paper 36[20] specifically describes where Generative AI is being used.
16
Competency 4. [General vs. Narrow AI] aims to address the distinction between
general AI and narrow AI. General AI refers to systems with the ability to perform
any intellectual task that a human can, while narrow AI is designed to handle specific
tasks and lacks the generalization capabilities of a human mind. However distinguishing between general and narrow AI can help learners grasp the current capabilities
of AI and the challenges that lie ahead in achieving more advanced forms of artificial intelligence,in total, 0 out of 48 papers addressed this competency (kappa: 0).
This absence of coverage suggests that distinguishing between general and narrow AI
might not be a primary focus in the current discourse on AI literacy, which could be
due to factors like: the practical focus of AI literacy on current narrow AI applications, the speculative and complex nature of general AI theories, and the possibility
that this differentiation is an emerging topic not yet fully integrated into educational
frameworks.
4.3 What AI can do?
Competency 5. [...]
Competency 6. [Imagine future AI] The competency ”Imagine Future AI” focuses
on the ability to envision and anticipate the advancements and applications of artificial intelligence in the coming years. It involves foreseeing novel AI technologies,
analyzing emerging trends, considering ethical and societal implications, and developing strategic plans to leverage AI for competitive advantage. This competency also
encompasses anticipating future challenges and devising innovative solutions, as well
as understanding the integration of AI with other technologies and disciplines to
drive progress and innovation. 12 out of 48 articles (24%) were identified as addressing Competency 6, especially those with Article-IDs: 6 [6], 12 [9], 14 [36], 15 [16],
17 [37], 19 [27], 25 [38], 33 [19], 37 [32], 38 [21], 39 [33], and 45 [12].. The Cohen’s
Kappa coefficient was calculated to be k= 0.62, indicating a substantial agreement
between Rater1 and Rater2. The focus here is on AI knowledge and its effects on
human-AI collaboration. The article explores how AI knowledge can enhance task
performance but also reduce the intention to continue using AI, highlighting the need
for context-specific AI education. It suggests that better understanding the nuances
of AI knowledge can lead to more effective human-AI collaboration and proposes
future research directions to optimize AI education and upskilling programs. The
article 25 outlines a curriculum designed to engage students with AI concepts through
practical and ethical explorations. It emphasizes the importance of exposing students
to AI-enhanced careers and real-world applications of AI. The curriculum includes
interactive simulations and projects that help students understand AI mechanisms,
recognize their strengths, and envision future career paths in AI-related fields.
4.4 How does AI work?
Competency 7. [(Knowledge) Representations] Knowledge needs to be modeled so
that machines are capable of understanding it. This is done using Knowledge Representations, which is an integral part of every AI system. Understanding the concepts
17
of Knowledge Representation is essential to grasp how computers possess knowledge
and to understand how Knowledge gets lost in representations. This competency has
been addressed by 10 out of 48 papers (around 20%) with an inter-rater agreement of
0.51 kappa-value, which indicates moderate agreement. [23] [7] [24] [15] [26] [37] [27]
[30] [35] [22]
Papers [23], [7], [24] and [15] focus on how to teach this competency through the use
of a prototype activity called ’knowledge net’. This activity also focuses on teaching
users that real world knowledge will get lost in knowledge representations.
On the other hand, Papers [26] [27] [30] [35], focus on how to enable teachers to better
teach AI literacy competencies like Knowledge Representations through examining
existing frameworks for teaching AI competencies or suggesting new ones. Another
Paper [37], also suggests that teaching Knowledge Representations to users will make
users delegate tasks to AI more efficiently, which we think is an important skill to
have in a world were people collaborate more and more with AI systems.
Competency 8. [Decision-Making] Competency ”Decision Making” is aims to to
recognize and describe examples of how computers reason and make decisions. Of the
48 papers reviewed, 9 papers specifically focused on this competency, that takes 19%
of all the papers, showing an inter-rater agreement with a kappa-value k=0,4 (The
nine papers are 3 [7], 9 [15], 28 [39], 32 [18], 33 [19], 34 [31], 40 [40], 48 [22], 50 [41]).
Paper 3[7], for example, the point ”Creature Features,” discussed in Section 3.2 of the
article, directly explores how agents represent knowledge and make decisions, which
are central aspects of this competency. In ”Creature Features” it mentions “learners
can use a card deck and “weight tokens” to provide training data to a feature-based
machine learning algorithm that classifies animals as birds.” This point is specifically
intended to communicate competencies related to decision-making in AI, by letting
participants adjust the training data and observe changes in how the algorithm classifies information based on those adjustments. The article discusses how this setup allows
learners to understand the impacts of their decisions on the algorithm’s performance,
aligning directly with Competency 8’s focus on understanding AI decision-making
processes. In a similar side, paper 48 [22]touches on AI Literacy’s Competency 8,
which is centered on how AI systems make decisions, particularly through the lens of
explainable AI (XAI). Explainable artificial intelligence (XAI) is a set of processes and
methods, with these allows human users to comprehend and trust the results created
by machine learning algorithms. From the paragraphs “BACKGROUND Explainable
artificial intelligence (XAI)” it shows the following statements: “In much of the ML
literature, XAI aims to make the reasons behind a ML model’s decisions comprehensible to humans.” This direct statement highlights how AI systems reason and make
decisions, fitting squarely with Competency 8’s requirements. From introduction part,
it mentions, “To close the gap between XAI algorithms and user needs for effective
transparency, the HCI community has called for interdisciplinary collaboration and
user-centered approaches to explainability.” This suggests a focus on how AI’s decisionmaking processes can be made understandable and meaningful to users, aligning with
the need for recognizing and describing examples of AI decision-making.
We believe it is crucial for Competency 8 [Decision-Making].
18
Competency 9. [Machine Learning steps] ML is not only the AI sub-field but
also overlaps each other. AI is the intersection of cognitive systems, robotics, and
machine learning [14]. Some recent research focuses on how to teach ML to individuals without a CS background and to non-experts to perform ML algorithms so that
they won’t be surprised that ML needs human decision-making. Having students
physically display algorithms to avoid misunderstanding learners should know computers learn from their data.[26]
Undergraduate and non-computer science students need to learn some specific ML
competencies and how ML techniques and their main algorithms work.
Advanced ML like neural networks and deep learning is beneficial for students. It can
be adopted in the political science domain, and medical students should understand
how AI systems can be introduced in their work and distinguish their responsibilities
from the machine’s. Journalist students need to understand AI ethical concerns.
Data literacy is really important when it comes to understanding machine learning.
You can’t really understand AI, especially from a machine learning perspective,
without being data literate. In the evaluation stage, it’s important for students to be
able to train a model and get how biases in the training data can affect the accuracy
and strength of a machine learning model.[10]
Competency 10. [Human Role in AI] is indispensable, as they are fundamental
to its operation. Humans creates datasets, has an important role on every activity,
set ethical guidelines and provide the creativity and critical thinking necessary to
innovate and address complex problems that AI aims to solve.
Furthermore, humans are essential in training AI models, such as choosing and working with available data [16] and, as they evolve, human oversight remains crucial to
monitor, interpret, and guide these technologies, ensuring they serve society’s best
interests and reflect human values. As for the future generations, since AI is increasingly present nowadays and will be used even more in the future, there is an increasing
need to prepare them for the future workforce and everyday life, for example, with
Machine Learning (ML), that can revolutionize practices of knowledge discovery [28].
Although it seems to be completely positive, when you think about how AI works,
you can see that the role of humans can actually be negative. As discussed in [23],
people can teach AI false information and, in the study in this article, the participating groups used a bad dataset to understand the consequences of this.
Since it is a competency strongly related to AI, after a 4-step verification, it could be
concluded that 17 of the 48 studied articles addressed this competency, 35% of them
all. The inter-rater agreement (kappa-value) on the third step was 0.26 and then the
last verification was made to get to the final results, since kappa-value was lower then
0.4.
Therefore, considering the positive and negative influence of humans in AI, the
transmission of humanistic ideas remains an important function of education. The
development of AI can not ignore the essence of education [39]
Competency 11. [Data literacy] The competency ”Data Literacy” focuses on the
understanding, evaluation, and effective use of data. This competency encompasses
19
skills in data acquisition, processing, analysis, and interpretation, alongside ethical
data management and visualization. Of the 48 papers reviewed, 4 papers were identified as addressing Competency 11, accounting for 8.3% of all papers, with an inter-rater
agreement kappa value of 0.13, indicating a low level of agreement between Rater 1
and Rater 2. (The four papers are 7[14], 16[26], 27[42], and 47[35]).
For instance, paper 7[14]demonstrates how to effectively teach data literacy in
primary and secondary education through the design and implementation of the ”ArtBot” game. Specifically, through hands-on practice and reflection, students gain a deep
understanding of and ability to apply data analysis skills. Furthermore, by helping
students recognize how human values and biases are reflected in AI applications, the
game fosters ethical awareness and a sense of responsibility in data handling.
Similarly, the review results of paper 16[26] show that existing AI literacy frameworks either barely cover or do not cover competencies related to data literacy at all.
However, data literacy is crucial for understanding AI, especially machine learning.
To address this gap, the article suggests that the frameworks be expanded to more
comprehensively promote both AI and data literacy.
Competency 12. [Learning from Data] It’s crucial for learners to understand that
AI gains knowledge from the data it receives.Understanding how computers learn
from their data is crucial for critically reviewing and interpreting machine learning
models, discovering and mitigating bias, supporting ethical AI practices, and involving learners in useful and applicable ways. This fundamental knowledge ensures that
learners are more than passive AI users; they are informed and critical thinkers able
to responsibly navigate and shape the AI-driven environment. Being able to identify
when their data is utilized to teach machine learning is an important aspect of data
literacy in the context of AI[1]. In total, 14 out of 48 papers (29%) addressed this
competency. The inter-rater agreement is 0.37, which indicates fair agreement.
For example, Paper ”Learn to Machine Learn: Designing a Game Based Approach
for Teaching Machine Learning to Primary and Secondary Education Students”[14],
through the implementation of the game ArtBot in the LearnML project, helps users
understand that AI gains knowledge from the data. ArtBot allows children to engage
with AI programming interactively. By manipulating data inputs and observing the
outcomes, learners can see firsthand how AI systems use data to learn and make
decisions.
Similarly, the paper ”Artificial Intelligence in Compulsory K-12 Computer Science
Classrooms: A scaleable Professional Development Offer for Computer Science
Teachers”[22] focuses on developing and implementing a professional development
(PD) program to prepare computer science (CS) teachers to teach artificial intelligence (AI) in K-12 education. The professional development (PD) program for K-12
computer science teachers contains learning objectives in machine learning and data
handling. The program guarantees that teachers comprehend how AI systems use
data to learn and judge by teaching these subjects. The progress test developed for
evaluating teachers includes questions regarding various machine-learning approaches
and algorithmic bias. Addressing algorithmic bias in the PD program emphasizes the
relevance of data quality and features while training AI systems. Teachers learn that
20
biases in data can result in biased AI outputs, emphasizing the importance of data in
AI learning processes. This emphasis helps teachers understand that data is essential
for training AI models and determines their behaviour and performance.
Competency 13. [...]
Competency 14. [...]
Competency 15. [Sensors] focuses on the role of sensors in AI systems. Learners
should understand the definition and functions of sensors in AI, identify various
types of sensors, comprehend the data representations and reasoning processes they
support, and recognize and explain sensors in everyday devices. Additionally, learners
should grasp how sensors integrate with AI algorithms to interpret data, make decisions, or perform actions. This competency is essential for a thorough understanding
and application of AI technology.
Out of 48 reviewed papers, four of them specifically addressed this competency,
the paper with competency ”Sensors” takes part of 8.3% of all 48 papers. With an
inter-rater agreement kappa value of 0.38 (the four papers ID are 9[15], 13[13], 19[27],
50[41]).
For instance, paper 9 [15]discusses the importance of understanding sensors, their
integration with AI, and their roles in decision-making and action, which aligns closely
with the described competency. From paragraph 8.2.7 Integrating technology of paper
9 we can find following statements:
”Sensors and actuators in particular seemed to elicit a lot of participant interest/dialogue throughout the workshop. The success of the paper-based prototypes at
facilitating learning dialogue is promising for the development of ’unplugged’ AI
learning experiences for low-cost learning interventions” .
These words addresses the role of sensors in AI systems, highlighting their importance in educational contexts and their ability to engage learners. This aligns with the
competency’s focus on understanding the functions of sensors and their types, further
more it can also show how they integrate with AI to support data interpretation and
decision-making processes.
Paper 50 [41] explores user involvement in the learning of intelligent agents, describing how smart home systems can associate sensor information with user-provided
activity labels to recognize user activities, thereby enhancing users’ understanding and
control over Al systems.
”Participants were briefly explained that the smart home would connect the information on which activity is being executed with sensory data of motion sensors and
smart devices in the home. This way, it would later be able to tell from similar sensory information which activity is executed by the resident. In fact, our prototype did
not yet actually collect any sensory information. Instead, we had fake motion sensors
installed in the apartment to make the prototypical smart home look fully realistic.”
As the words comes in 4.1 Method, it describes how the smart home system integrates sensory data to recognize and react to user activities. It addresses the definition
and function of sensors in AI, their role in data collection and interpretation, and
their application in everyday devices, which are key components of Competency 15.
21
4.5 How should AI be used?
Competency 16. [Ethics] The rapid advancement of AI has raised critical ethical
questions. From potential biases in decision-making algorithms to concerns about data
privacy, ensuring responsible AI development is essential. In total, 17 papers out of
48 (35,4%) addressed ethical inquiries. For a more comprehensive overview, a table is
provided to summarize the frequency of individual concepts
Concept Frequency Percentage
Bias 12 25%
Ethical decision-making 5 10%
Fairness 5 10%
Data privacy 5 10%
Misinformation 4 8%
Diversity 2 4%
Discrimination 2 4%
Transparency 2 4%
Accountability 1 2%
Table 7: Breakdown of concepts and their frequency
Fig. 4: AI ethics word cloud
Bias: Papers IDs 1 [23], 3 [7], 4 [8], 7 [14], 9 [15], 10 [25], 11 [43], 26 [44], 28 [39],
29 [17], 33 [19], 35 [45]. Most papers highlighted the existence of algorithmic biases.
These biases are frequently associated with the inherent biases present in the training dataset. Paper 10 [25] pointed out that there is increasing concern about how the
opacity and complexity of ML systems can result in systemic racial and gender biases.
Additionally, paper 35 [45] showed that it is necessary to understand accessibility in
relation to AI, as most AI courses often neglect disability.
Ethical decision making: Papers IDs 7 [14], 9 [15], 10 [25], 28 [39], 29 [17]. In addition
to the advantages and benefits of AI, there are concerns about the potential drawbacks
22
and challenges related to the decisions these systems might make, the autonomy of
AI agents, and the values and biases built into their design [14]. Paper 28 [39]pointed
out that the current ML technologies have their limitations, such as their tendency
to reflect and reinforce existing societal biases and discrimination, their potential to
pick up spurious correlations rather than causal relationships, and their performance
in generalizing to external data. The inclusion of diverse human perspectives in the
training of AI models used for critical decision-making, especially those that disproportionately impact marginalized groups is needed[15].
Fairness: Papers IDs 5 [46], 26 [44], 33 [19], 34 [31], 35 [45]. In technical AI/ML courses,
fairness is a frequently discussed topic, often paired with bias. The primary focus of
these courses is on promoting fairness in AI systems [44]. Paper 33 [19] introduces
Let’s Chance, a novel computational microworld extending the Scratch Programming
Language to help children explore the ethical issue of AI, including fairness. Paper 34
[31] shows how art can offer alternative approaches to developing public AI literacy,
focusing on fairness by providing creative and tangible interactions to better understand AI processes beyond technical methods.
Data privacy: Papers IDs 3 [7], 6 [6], 9 [15], 10 [25], 26 [44]. Increasing worries regarding AI’s contribution to misinformation, and data privacy violations underscore the
necessity for technology users to acquire new skills to engage with AI more critically
and thoughtfully. Paper 6 [6] introduces the AI Educator Make-a-Thon as a model
that enhances educators’ engagement and investment in AI literacy. To engage teachers with new perspectives on AI technologies, expert panels regarding data privacy and
other issues were organized. Paper 26 [44] examines the integration of ethics into AI
education through standalone courses and embedding ethics with technical AI courses
at US universities. Privacy was specifically addressed in 32% of the courses, often in
combination with other topics such as bias, regulation, and automation. Additionally,
two courses included at least one lecture on differential privacy.
Misinformation: Papers IDs 1 [23], 3 [7], 4 [8], 9 [15]. Recent studies highlight
widespread public concerns regarding AI in everyday applications, such as the
propagation of viral misinformation [15]. Misinformation is the consequence of misunderstanding or inadequately regulating AI’s decisions [23].
Diversity: Papers IDs 9 [15], 14 [36]. The new advances in AI technology can help
people from disadvantaged groups to better integrate and communicate with others.
For example paper 14 [36] looks into teachers’ perspective on the implementation of
an AI education curriculum and how it can stimulate reflections around inclusion and
diversity.
Discrimination: Papers IDs 3 [7], 28 [39]. Similarly as in Misinformation explained
AI can be used with malicious intent to discriminate others. Moreover the AI-models
themself can be discriminatory, if not properly supervised. To get a deeper insight
into the reliance of laypeople on ML models, paper 28 [39] studies the effects of user
tutorials on this subject.
Transparency: Papers IDs 10 [25], 27 [42]. To better understand how to use a particular AI it is important to clearly and transparently communicate to the users the
capabilities and limitations of such systems. To study this aspect, paper 27 [42] looks
23
at how children come to see machine intelligence after training custom machine learning models and creating smart programs that use them.
Accountability: Papers ID 10 [25]. Research driven by AI ethical concerns focuses
on increasing accountability in ML models by improving transparency, fairness, and
robustness, and ensuring that ML system designers communicate and justify their
decisions to make them accountable for their impact on broader social systems.
4.6 How do people perceive AI?
Competency 17. [Programmability]
Understand that agents are programmable is foundational for understanding
how AI works [1], since it is built and controlled by human-written code and logic.
However, it is important for designers to keep in mind that prerequisite coding skills
can be a barrier to entry into the AI world. [1]. In total, 12 out of 48 papers (25%)
addressed this competency. The inter-rater agreement is 0.38, which indicates fair
agreement (Paper IDs 1[23], 7[14], 11[43], 12[9], 14[36], 27[42], 31[30],33[19], 39[33],
45[12], 47[35], 50[41]).
Paper IDs 12[9] discusses that there is a clear need for developing and implementing
separate courses that are specifically designed for various segments of the public. In
this way, all citizens will learn about AI and the impact intelligent algorithms have
on their lives, at a level that they can actually understand and relate to without
necessarily having a technical background. However, understanding what exactly is
needed when designing AI education for the broad public is an important part of the
process, and to our best knowledge, there is still work to be done.
Most of the papers studied mentioned children’s perceptions of AI. Since they are
future users of it, it is crucial to consider their opinions so its use can be more accessible and effective in the future. There are multiple factors that influence children’s
views of AI: age, gender, cultural background (since it can lead to lack of prior
experience) and more. The form of AI might make a difference in perception as well.
Some works can contribute to understanding how different types of AI technologies
can influence youth users’ understanding and judgments about fairness. [43]
Paper ”What is AI Literacy? Competencies and Design Considerations” [1] mentions
that children have different ways of perceiving things compared to adults, such as their
tendency to focus on observable characteristics (e.g., success) rather than unobservable ones (e.g., strategy) when assessing AI. This makes it challenging to help users
not only understand but also know how to apply AI. Paper ”Learn to Machine Learn:
Designing a Game Based Approach for Teaching Machine Learning to Primary and
Secondary Education Students”[14] implemented a game called ArtBot, which allows
children to play while practicing AI programming in a guided manner, enhancing their
attention. Their goal in the LearnML project and through the game ”ArtBot” is to
develop educational AI and ML support materials while stimulating students’ critical
thinking about the aspects, factors, and biases that can shape AI agents and systems.
24
5 Discussion
• What was our original goal for this work and what did we find?
• How do our findings compare with the original research of Long&Magerko?
• How do our findings relate to other frameworks of AI Literacy?
• What are the theoretical and practical implications of our work?
6 Conclusion
• A brief summary about what we talked about in the paper
• What are the limitations of this research? A limitation could be that our dataset
comprised English-language articles,that potentially excludes relevant studies published in other languages that lead our findings to not fully capture the global
research perspectives on the topic.
• What are potential directions for future work?
References
[1] Long, D., Magerko, B.: What is ai literacy? competencies and design considerations. In: Proceedings of the 2020 CHI Conference on Human Factors in
Computing Systems, pp. 1–16 (2020)
[2] Arksey, H., O’Malley, L.: Scoping studies: towards a methodological framework.
International Journal of Social Research Methodology 8(1), 19–32 (2005) https:
//doi.org/10.1080/1364557032000119616
[3] al., K.M.: Understand, evaluate, and use emerging technology. Digital Promise
(1), 40 (2024)
[4] al., U.-D.E.: About (2022). Accessed 2024-06-26
[5] Casal-Otero, L., Catala, A., Fern´andez-Morante, C., Taboada, M., Cebreiro, B.,
Barro, S.: Ai literacy in k-12: a systematic literature review. International Journal
of STEM Education 10(1), 29 (2023)
[6] Sperling, K., Stenberg, C.-J., McGrath, C., ˚Akerfeldt, A., Heintz, F., Stenliden,
L.: In search of artificial intelligence (ai) literacy in teacher education: A scoping
review. Computers and Education Open, 100169 (2024)
[7] Long, D., Padiyath, A., Teachey, A., Magerko, B.: The role of collaboration,
creativity, and embodiment in ai learning experiences. In: Proceedings of the 13th
Conference on Creativity and Cognition (2021)
[8] Lee, I., Zhang, H., Moore, K., Zhou, X., Perret, B., Cheng, Y., Zheng, R., Pu, G.:
AI Book Club: An Innovative Professional Development Model for AI Education,
pp. 202–208 (2022)
25
[9] Kasinidou, M., Kleanthous, S., Otterbacher, J.: Artificial intelligence in everyday
life: Educating the public through an open, distance-learning course. In: Proceedings of the 2023 Conference on Innovation and Technology in Computer Science
Education V. 1, pp. 306–312 (2023)
[10] Ten´orio, K., Romeike, R.: Ai competencies for non-computer science students in
undergraduate education: Towards a competency framework. In: Proceedings of
the 23rd Koli Calling International Conference on Computing Education Research
(2024)
[11] Khadivi, P.: Ai and society: Teaching ai to non-stem students, pp. 18–27 (2023)
[12] Song, Y., Tian, X., Regatti, N., Katuka, G.A., Boyer, K.E., Israel, M.: Artificial intelligence unplugged: Designing unplugged activities for a conversational
ai summer camp. In: Proceedings of the 55th ACM Technical Symposium on
Computer Science Education V. 1, pp. 1272–1278 (2024)
[13] Long, D., Blunt, T., Magerko, B.: Co-designing ai literacy exhibits for informal
learning spaces. In: Proceedings of the ACM on Human-Computer Interaction,
pp. 1–35 (2021)
[14] Voulgari, I., Zammit, M., Stouraitis, E., Liapis, A., Yannakakis, G.N.: Learn to
machine learn: Designing a game based approach for teaching machine learning
to primary and secondary education students. In: IDC ’21: Proceedings of the
20th Annual ACM Interaction Design and Children Conference (2021)
[15] Long, D., Blunt, T., Magerko, B.: Co-designing ai literacy exhibits for informal
learning spaces. Proc. ACM Hum.-Comput. Interact. (2021)
[16] Tseng, Y., Xiao, R., Bogart, C., Savelka, J., Sakr, M.: Assessing the efficacy of ˇ
goal-based scenarios in scaling ai literacy for non-technical learners. In: SIGCSE
2024: Proceedings of the 55th ACM Technical Symposium on Computer Science
Education V. 2 (2024)
[17] Schaper, M., Malinverni, L., Valero, C.: Robot presidents: Who should rule
the world?: Teaching critical thinking in ai through reflections upon food
traditions. In: NordiCHI ’20: Proceedings of the 11th Nordic Conference on
Human-Computer Interaction: Shaping Experiences, Shaping Society (2020)
[18] Wan, X.: A study on the current development of artificial intelligence in education industry in china. In: ICETT ’21: Proceedings of the 2021 7th International
Conference on Education and Training Technologies (2021)
[19] Dhariwal, M., Dhariwal, S.: Let’s chance: Playful probabilistic programming for
children. In: CHI EA ’20: Extended Abstracts of the 2020 CHI Conference on
Human Factors in Computing Systems (2020)
26
[20] Nithithanatchinnapat, B., Maurer, J., Deng, X.N., Joshi, K.D.: Future business workforce: Crafting a generative ai-centric curriculum today for tomorrow’s
business education. ACM SIGMIS Database: the DATABASE for Advances
in Information Systems 55(1), 6–11 (2024) https://doi.org/10.1145/3645057.
3645059
[21] Wang, H.: Integration path of artificial intelligence technology and education
and risk avoidance. In: ICETM ’22: Proceedings of the 2022 5th International
Conference on Education Technology Management (2023)
[22] Liao, Q.V., Gruen, D., Miller, S.: Questioning the ai: Informing design practices
for explainable ai user experiences. In: Proceedings of the 2020 CHI Conference
on Human Factors in Computing Systems (2020)
[23] Long, D., Teachey, A., Magerko, B.: Family learning talk in ai literacy learning
activities. In: Proceedings of the 2022 CHI Conference on Human Factors in
Computing Systems (2022)
[24] Long, D., Rollins, S., Ali-Diaz, J., Hancock, K., Nuonsinoeun, S., Roberts, J.,
Magerko, B.: Fostering ai literacy with embodiment & creativity: From activity
boxes to museum exhibits. In: Proceedings of the 22nd Annual ACM Interaction
Design and Children Conference. IDC ’23. ACM, ??? (2023). https://doi.org/10.
1145/3585088.3594495 . http://dx.doi.org/10.1145/3585088.3594495
[25] Bilstrup, K.E.K., Kaspersen, M.H., Assent, I., Enni, S., Petersen, M.G.: From
demo to design in teaching machine learning. In: FAccT ’22: Proceedings of the
2022 ACM Conference on Fairness, Accountability, and Transparency (2022)
[26] Olari, V., Romeike, R.: Addressing ai and data literacy in teacher education: A
review of existing educational frameworks. In: Proceedings of the 16th Workshop
in Primary and Secondary Computing Education (2021)
[27] Polak, S., Schiavo, G., Zancanaro, M.: Teachers’ perspective on artificial intelligence education: an initial investigation. In: IUI ’22: Proceedings of the 27th
International Conference on Intelligent User Interfaces (2022)
[28] Wan, X., Zhou, X., Ye, Z., Mortensen, C.K., Bai, Z.: Smileycluster: supporting
accessible machine learning in k-12 scientific discovery. In: Proceedings of the
Interaction Design and Children Conference. IDC ’20. ACM, ??? (2020). https://
doi.org/10.1145/3392063.3394440 . http://dx.doi.org/10.1145/3392063.3394440
[29] Stephanie Tena-Meza, A.A. Miroslav Suzara: Coding with purpose: Learning ai in
rural california. In: ACM Transactions on Computing EducationVolume 22Issue
3 (2022)
[30] Quiroz, P., Gutierrez, F.J.: Scratch-nb: A scratch extension for introducing k12 learners to supervised machine learning. In: Proceedings of the 55th ACM
27
Technical Symposium on Computer Science Education V. 1 (2024)
[31] Hemment, D., Currie, M., Bennett, S., Elwes, J., Ridler, A., Sinders, C., Vidmar,
M., Hill, R., Warner, H.: Ai in the public eye: Investigating public ai literacy
through ai art. In: 2023 ACM Conference on Fairness, Accountability, and Transparency. FAccT ’23. ACM, ??? (2023). https://doi.org/10.1145/3593013.3594052
. http://dx.doi.org/10.1145/3593013.3594052
[32] Guerzhoy, M., Neumann, M., Virtue, P., Anderson, C.J., Singla, Y.K., Orchard,
A., Shah, R.R.: Eaai-23 blue sky ideas in artificial intelligence education from
the aaai/acm sigai new and future ai educator program. AI Matters 9(2), 24–29
(2023)
[33] Thomas, G.: Redesign of an elective introductory artificial intelligence course in
a credit-limited computer science curriculum. Journal of Computing Sciences in
Colleges 37(2), 98–104 (2021)
[34] Jianlei Yang, W.Z. Xiaopeng Gao: Towards systems education for artificial intelligence: A course practice in intelligent computing architectures. In: GLSVLSI
’20: Proceedings of the 2020 on Great Lakes Symposium on VLSI (2020)
[35] Jetzinger, F., Baumer, S., Michaeli, T.: Artificial intelligence in compulsory k-12
computer science classrooms: A scalable professional development offer for computer science teachers. In: Proceedings of the 55th ACM Technical Symposium
on Computer Science Education V. 1 (2024)
[36] Gibellini, G., Fabretti, V., Schiavo, G.: Ai education from the educator’s perspective: Best practices for an inclusive ai curriculum for middle school. In: Extended
Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems,
pp. 1–6 (2023)
[37] Pinski, M., Adam, M., Benlian, A.: Ai knowledge: Improving ai delegation
through human enablement. In: Proceedings of the 2023 CHI Conference on
Human Factors in Computing Systems (2023)
[38] Lee, I., Ali, S., Zhang, H., DiPaola, D., Breazeal, C.: Developing middle school
students’ ai literacy. In: Proceedings of the 52nd ACM Technical Symposium on
Computer Science Education, pp. 191–197 (2021)
[39] Chun-Wei Chiang, M.Y.: Exploring the effects of machine learning literacy
interventions on laypeople’s reliance on machine learning models. In: Extended
Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems
(2022)
[40] Jakob Schoeffer, Y.M. Niklas Kuehl: “there is not enough information”: On the
effects of explanations on perceptions of informational fairness and trustworthiness in automated decision-making. In: FAccT ’22: Proceedings of the 2022 ACM
28
Conference on Fairness, Accountability, and Transparency (2022)
[41] Sieger, L.N., Hermann, J., Schom¨acker, A., Heindorf, S., Meske, C., Hey, C.-
C., Do˘gang¨un, A.: User involvement in training smart home agents: Increasing
perceived control and understanding. In: Proceedings of the 10th International
Conference on Human-Agent Interaction. HAI ’22. ACM, ??? (2022). https://doi.
org/10.1145/3527188.3561914 . http://dx.doi.org/10.1145/3527188.3561914
[42] Druga, S., Ko, A.J.: How do children’s perceptions of machine intelligence change
when training and coding smart programs? In: IDC ’21: Proceedings of the 20th
Annual ACM Interaction Design and Children Conference (2021)
[43] Solyst, J., Xie, S., Yang, E., Stewart, A.E.B., Eslami, M., Hammer, J., Ogan, A.:
“i would like to design”: Black girls analyzing and ideating fair and accountable
ai. In: Proceedings of the 2023 CHI Conference on Human Factors in Computing
Systems. CHI ’23. ACM, ??? (2023). https://doi.org/10.1145/3544548.3581378 .
http://dx.doi.org/10.1145/3544548.3581378
[44] Garrett, N., Beard, N., Fiesler, C.: More than” if time allows” the role of ethics in
ai education. In: Proceedings of the AAAI/ACM Conference on AI, Ethics, and
Society, pp. 272–278 (2020)
[45] Tseng, C.E., Jung, S.H., Elglaly, Y.N., Liu, Y., Ludi, S.A.: Exploration on integrating accessibility into an ai course. In: SIGCSE 2022: Proceedings of the 53rd
ACM Technical Symposium on Computer Science Education - Volume 1 (2022)
[46] Xie, S., Solyst, J., Ogan, A., Hammer, J.: Booklet-based design fiction to support
ai literacy. In: Proceedings of the 54th ACM Technical Symposium on Computer
Science Education V. 2. SIGCSE 2023. ACM, ??? (2022). https://doi.org/10.
1145/3545947.3576248 . http://dx.doi.org/10.1145/3545947.3576248
29




// I want to answer the question: • What is the motivation for doing this work? Why is this work important?
//But i dont really know. Give me some ideas