<figure><img src="image_1.png" /></figure>

AI4T
AI FOR TEACHERS

Work Package 2:

**Training architecture and resources**

**Deliverable D2.3**

**Template for AI characterization**

<figure><img src="image_2.png" /></figure>

Co-funded by the
Erasmus+ Programme
of the European Union

<figure><img src="image_3.png" /></figure>
<table><tr><td>Project Title:</td><td>Artificial Intelligence for and by Teachers</td></tr><tr><td>Project Acronym:</td><td>AI4T</td></tr><tr><td>Project Number</td><td>626154</td></tr><tr><td>Grant Agreement Number</td><td>626154-EPP-1-2020-2-FR-EPPKA3-PI-POLICY</td></tr><tr><td>Instrument:</td><td>Erasmus Plus, Key Action 3</td></tr><tr><td>Deliverable number</td><td>D2.3</td></tr><tr><td>Work Package</td><td>2 Training architecture and resources</td></tr><tr><td>Work Package Leader</td><td>University of Nantes</td></tr><tr><td>Work Package Partners</td><td>NRIA, Université de Lorraine, MENJS, MIZS, University of Maribor, H2, DWEC, SCRIPT, ITD-CNR, MI, FEI</td></tr><tr><td>Dissemination level</td><td>Public</td></tr><tr><td>Delivery date</td><td>2022-11</td></tr><tr><td>Status</td><td>Final</td></tr><tr><td>Version</td><td>6.0</td></tr><tr><td>Author(s)</td><td>Jiajun PAN, Azim ROUSSANALY, Anne BOYER</td></tr><tr><td>List of contributors</td><td>All partners</td></tr><tr><td>Deliverable Manager / Reviewers</td><td>France Education International</td></tr></table>

Disclaimer

This project has been funded with support from the European Commission. This publication [reflects the views only of the author, and the Commission cannot be held responsible for any use which may be made of the information contained therein.

<figure><img src="image_4.png" /></figure>
<figure><img src="image_5.png" /></figure>

ABSTRACT

In the AI4T project, we found that teachers lacked knowledge of AI tools, and in another WP2 outcome, "How Al can support teachers at their job?", we summarised the problems teachers face when using Al at work, and pointed to a lack of a standardized template that is objective and credible. We therefore propose a 7-layer template for comprehensively presenting the features of AI tools, helping users who lack relevant knowledge to understand these tools, and enhancing communication between teachers and Al tool publishing companies.

Therefore, in this paper, we present the 7 layers of the template in order: Usage; Decision level; Personal data; Data security; Transparency; Technology; Algorithm and introduce the definition of each layers. Each layer of the template is designed to answer a particular aspect of a teacher's query.

In order to investigate these Al educational resources and represent them in our proposed templates, we designed a questionnaire for publishers and companies of Al educational resources. The questionnaire helps us to fill out the templates more easily based on the connection between the questions and the levels in the templates. The questionnaire and its design can be used not only for the AI4T project, but also for other projects investigating or evaluating Al educational resources. To make it easier to communicate the features of the Al tool directly to teachers, we also designed a leaflet that simplifies the template.

Finally we showed the template to a number of Al companies and filled it in through questionnaires and interviews.

KEYWORDS

Al in Education; Template of AI tools

Dissemination level

<table><tr><td>PU</td><td>Public</td><td>X</td></tr><tr><td>PP</td><td>Restricted to project partner (including the Commission)</td><td></td></tr><tr><td>RE</td><td>Restricted to a group defined by the consortium (including the Commission)</td><td></td></tr><tr><td>CO</td><td>Confidential, only for members of the consortium (including the Commission)</td><td></td></tr></table>

<figure><img src="image_6.png" /></figure>
<figure><img src="image_7.png" /></figure>

Table of contents

<table><tr><td>Introduction</td><td>4</td></tr><tr><td>1. Background: why we need to propose the template</td><td>7</td></tr><tr><td>2. Definition of the template: what kind of template do we propose</td><td>10</td></tr><tr><td>3. Methodology for the questionnaire: how should we design the questionnaire</td><td>16</td></tr><tr><td>4. Leaflet and template: key information directly shown to teacher</td><td>21</td></tr><tr><td>5. Example of the template: how should we use the template</td><td>28</td></tr><tr><td>6. Reference</td><td>38</td></tr></table>

<figure><img src="image_8.png" /></figure>
<figure><img src="image_9.png" /></figure>

AI4T
AI FOR TEACHERS

# Introduction

This document has been prepared as part of the European AI4T project and is a deliverable of WP2 – Training architecture and resources. Another delivery of WP2 is the Synthesis on AI Education report titled "How AI can support teachers at their job?".

In that report, we summarize the issues that teachers face when using AI in their work and note the lack of objective and credible standard templates that can comprehensively demonstrate the characteristics of AI tools and help users who lack relevant knowledge to understand them.

<figure><img src="image_10.png" /><figcaption>Figure 1: The relation between this paper and 'How AI can support teachers at their job?' report.</figcaption></figure>

The purpose of this work is to provide a template that demonstrates the attributes associated with the use of artificial intelligence (AI) technology in educational resources. The template therefore aims to link each resource to a form generated from an instance of the proposed template and to help users understand the characteristics of the resource. In our case, the template is mainly intended for teachers who are selected by WP1- Experimentation in AI4T and who are already aware of the use of AI in the educational resources proposed to them.

For the purpose of this study, an AIER (AI Educational Resource) has a broad meaning and can be an educational platform or a standalone or online application. To investigate these AIERs and represent them in our proposed template, we designed a questionnaire for the creators and companies of AIERs. This questionnaire and its design of it can not only be used in the AI4T project but also be suited to other projects which investigate or evaluated AI educational resources.

This document will be organized as follows:

<figure><img src="image_11.png" /></figure>
<figure><img src="image_12.png" /></figure>

* In the background section, we present the status of AI education resources and our reasons for proposing templates. And the focus and limitations of this template are clarified.

* In the definition section, we define a template with 7 layers. For each layer of the template, we introduce its meaning and necessity and give examples.

* In the questionnaire section, we present a method for designing questionnaires. And based on this method and the feedback from the subsequent practice, we also give the questionnaire we have designed.

* In the leaflet section, we explain the need for leaflets and the relationship and difference between templates and leaflets. The template is mainly used for research related to AI experts, educators and government workers, while the flyer, on the other hand, will be used to directly help teachers understand information about AIER.

* In the experimental section, we present several results of using templates on real-world AI tools to summarize their characteristics and information. We showed the templates to AI companies and filled them out with questionnaires and interviews.

<figure><img src="image_13.png" /></figure>
<figure><img src="image_14.png" /></figure>

# 1. Background

## Why we need to propose the template.

Artificial Intelligence in Education (AIEd) is an emerging area in education, which has been around for about 30 years. However, it is still unclear how AIEd can be fully leveraged to influence teaching and learning on a larger scale, and how it can be used to its pedagogical advantage. According to a survey of K12 education in Europe in 2021, 72% of countries are interested in AIEd, yet most are only beginning to experiment with AI involvement in education or have not even begun to do so. The survey by eLearning Industry predicts that over 47% of learning management tools will be AI-enhanced within the next three years [1][2].

It's important to note that when discussing AI in education, we refer only to AI used for educational tasks, not for general tasks in both educational and non-educational scenarios, such as AI translators. In this report, we focus on AI that is used only for educational tasks, which we denoted it as AIER (AI Educational Resource). AIER includes the AI resource using AI methods particularly designed for educational tasks, as well as the one using common AI methods. We will not limit the definition of AIER by the technology used.

AIER is the core of AIEd, with educational resources encompassing not only courses and textbooks but also online platforms, among other resources. AIERs are educational resources that work with AI technology. In this project, AI4T, we have noticed a lack of knowledge and trust in AIER among our main service group, including teachers, related groups, students, and educational institutions [8][9]. Additionally, existing companies that develop AIERs are more focused on the technical domain and commercial promotion of AIERs.

This leads to the main purpose of this paper: to help AIER publishers to present the full range of AIER features and to help users (especially teachers) who do not have the relevant knowledge to understand AIER.

<figure><img src="image_15.png" /></figure>
<figure><img src="image_16.png" /></figure>

Therefore, we propose a template that focuses on the characteristics of AI integration into educational resources at all levels: scientific, technical, regulatory, ethical, etc. In addition, the information we have compiled is intended to be as objective as possible. It is primarily intended to give teachers a better understanding of the resources they use or that their students use. It is not intended to categorize or compare the solutions proposed by the various resource publishers.

<figure><img src="image_17.png" /><figcaption>Figure 2: To help teachers and AI publishers make connections and improve understanding. We proposed templates and sent a questionnaire to AI publishers based on the templates.</figcaption></figure>

<figure><img src="image_18.png" /></figure>
<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /><figcaption>Figure 3: Using questionnaires filled out by AI publishers, we extracted information to fill out templates for the study. And based on the template we simplify it into a leaflet to give to the teacher.</figcaption></figure>

In order to maintain the comprehensiveness and professionalism of the template, we have tried to be as comprehensive as possible in the definition of the various layers of the template, which may lead to comprehension difficulties for both AI publishers and teachers. And in order to avoid inducing and biasing the readers in the responses of the AI publishers, and in order to allow the AI publishers and the teachers to understand each other better, on the basis of this template, we designed a questionnaire facing the AI publishers. As shown in Figure 2, this questionnaire will be provided to publishers by AI experts (in the AI4T project, this will be our work in WP2) in order to collect information about AIER from them. The details of this questionnaire are presented in the questionnaire section. As shown in Figure 3, after obtaining the answers to the questionnaire, the AI expert can fill in the template as a meaningful example. This example will be used on the one hand for AI4T project and research use by educators, and on the other hand, as shown in Figure 3, the example will be simplified and provided to teachers in the form of a leaflet to make it easier to understand.

It is worth noting that whilst the primary use of this template is for use within the AI4T project, and it has been designed primarily for selected teachers in AI4T. However, the template and the associated questionnaires and leaflets derived from it could also be used in other studies on AI tools. In the next section, we present the sources and methods for the design of the various layers of the template, so that following the design methodology of this template, it can be easily extended to other projects according to different needs and tasks.

<figure><img src="image_21.png" /></figure>
<figure><img src="image_22.png" /></figure>

## 2. Definition of the template

### What kind of template do we propose?

According to the four-part classification framework of AI technology [3], the four dimensions to classify AI are:

* **Context**: The environment where the system is being deployed and who is deploying it.

* **Data and Input**: The data the system uses and the kinds of input it receives.

* **AI Model**: The underlying particularities that make up the AI system.

* **Task and Output**: The tasks the system performs and the outputs that make up the results of its work.

In the case of AIERs, the basic characteristics to describe it should be the educational environment, the *input data from the user* (teacher, student or related group), the *AI technology* (including algorithms) and the educational task (*usage scenario*).

A survey of AIEd [4] shows a Technology-Organisation-Environment Framework, which denoted AIERs with 3 parts and 1 output. The 3 parts are technology, organization (*users and usage*), environment (*support and input data*). The output is the decisions. And the level of decision making that the user can decide in the final output is noteworthy. The *usage scenario and decision level* of the users have been pointed out again in [2].

In Explainable Artificial Intelligence in education [5], besides the characteristics mentioned before, another dimension is proposed for AIERs which is *transparency*. To increase trust in AI systems, it is necessary to promote the use of methods that generate transparent explanations and justifications for the decisions made by AI systems.

Another dimension to increase the trust in AI is *data security*, in [1][6], as for the AI4T project, it can see that concerns about data security are an important source of distrust in AI technology.

Not only to satisfy the requirement of teachers and related users to understand AIERs, but also based on the basic characteristics of AIERs, summarized by the dimensions mentioned before, we design a 3-class 7-layer template. The layers of template are the following:

<figure><img src="image_23.png" /></figure>
<figure><img src="image_24.png" /></figure>

<table><tr><td>Usage</td></tr><tr><td>Decision level</td></tr><tr><td>Personal data</td></tr><tr><td>Data security</td></tr><tr><td>Transparency</td></tr><tr><td>Technology</td></tr><tr><td>Algorithm</td></tr></table>

The first group is the Usage Scenario and the Decision level. This class is to clarify the purpose and users of an AIER and to help teachers understanding what their role is in using this AIER. The core questions to ask in these layers are as follows:

<table><tr><td rowspan="3">Usage</td><td>Who is this resource primarily intended for?</td></tr><tr><td>What is the intended use of this resource?</td></tr><tr><td>What area of teachers' work does this resource support?</td></tr></table>

Usage scenarios layer describes the users and applications of the AIER, which is often the first information teachers want to know. AIER could be classified into 3 types: Learner-oriented AI, Instructor-oriented AI and Institutional system-oriented AI [1]. There are many different usage scenarios based on the type of AIER. For example, a learner-oriented AIER could be an intelligent computer system designed to help students learn a course, while a teacher-oriented AIER could be one that automatically generates questions and tests for teacher-prepared courseware for a particular course.

<table><tr><td rowspan="4">Decision level</td><td>What is the type of output of the AI in the resource? Is it a decision, a recommendation, a proposal or a service?</td></tr><tr><td>What results does the AI give in the resource?</td></tr><tr><td>Who can see the result of this resource?</td></tr><tr><td>Can teachers access, modify, rewrite or otherwise influence the results given by the AI?</td></tr></table>

<figure><img src="image_25.png" /></figure>
<figure><img src="image_26.png" /></figure>

<table><tr><td></td><td>Are teachers involved in the production of the result? If so, how do they do this?</td></tr><tr><td></td><td>Who is responsible for the result?</td></tr></table>

Decision level layer describes the proportion of the final outcome that is attributable to the faculty when teacher using AIER. Normally, the decision level depends on the output of the AIER. For example, for AI that uses scoring strategies to automatically assess student learning outcomes, the level of decision making depends on whether the scores are sent directly to the student or to the teacher and who makes the subsequent assessment results. Decision level help teacher know their right and role in using an AIER.

The second group is Personal data, Security and Transparency. This class is to clarify the input data used by AIER and the secure handling of that data, to make it clear that AIER demonstrates the transparency of the internal mechanisms and to help teachers trust in the security of AIER. The core questions to ask in these layers are as follows:

<table><tr><td rowspan="3">Personal data</td><td>What personal data is collected for the needs of the AI technology in the resource?</td></tr><tr><td>What personal data is collected for other functions in addition to the AI technology in the resource?</td></tr><tr><td>Is data generated during interaction with the resource collected? If so, for what purpose?</td></tr></table>

Personal data layer outlines the personal data usage within AIER. As information security grows in importance, we focus on two key aspects in this layer:

1. Not all data collected by AIER is used exclusively for AI technology within AIER; some additional information serves other purposes such as creating access accounts. Clearly distinguishing between necessary data for AIER's functioning and optional additional services enables teachers to understand the data required and provides the option to refuse certain information for security reasons.

2. Apart from the explicitly provided data for AIER, external sources like shared social accounts may collect additional personal information. The creation of this layer is essential for teachers to gain clarity on this information.

<figure><img src="image_27.png" /></figure>
<table><tr><td rowspan="4">Data security</td><td>Is the personal data anonymous? If so, how does it work?</td></tr><tr><td>How the resource collects consent under local regulation?</td></tr><tr><td>Can the personal data used in this resource be audited by third parties? If so, what personal data can be audited and how can it be accessed?</td></tr><tr><td>What are the possibilities for outsiders to audit the data in this resource?</td></tr></table>

The data security layer specifies whether non-user visitors and external visitors to AIER can access the data and how it can be handled securely. This layer is related to the previous one. Different personal data should be handled in different ways, such as anonymity, encryption, and denial of access.

Anonymization means that the information recipient is completely unable to identify the data subject directly or briefly. There are five common data anonymization operations: generalization, suppression, anatomization, permutation, and perturbation [7]. Whereas encryption or also known as pseudonymization means that the information receiver cannot identify the data subject directly, but the information can be de-anonymized by the data manager after cross-identification with other information. Denial of access is a simple and straightforward way to keep the information only in the information manager, without giving access to others.

In this layer, it also clarifies whether the data will be used by external visitors. Many software share data with their own data partners nowadays, and if this is the case, it can also lead to different security of the data.

<table><tr><td rowspan="4">Transparency</td><td>What is the outcome or end service provided by this resource?</td></tr><tr><td>In what way is this result presented to the user?</td></tr><tr><td>Is there a website or other publicly available way to explain how this resource works? If so, what is it?</td></tr><tr><td>For the AI techniques used in the resources, are there any relevant published papers or documents on the above website or other accessible sources? If so, what is it?</td></tr><tr><td></td><td>Is the software used in this resource audible? If so, how does it work?</td></tr></table>

<figure><img src="image_28.png" /></figure>
<figure><img src="image_29.png" /></figure>

The Transparency Layer articulates two aspects of transparency: firstly, transparency into the inner workings of the AIER, i.e. its level of comprehensibility. This depends more on the way in which the educational task is accomplished and the results are presented. For example, if the educational task is to assess students through their learning behaviors, whether the AIER's is selecting certain learning behaviors and using certain parameters to generate the final report, and whether the selection of these learning behaviors and parameters is well documented, determines the transparency of this part. If the AIER is utilizing an AI algorithm for which it is difficult to explain the internal mechanisms, this part is less transparent.

Another component is the transparency of the AIER about its own information. This depends on the AIER publisher's presentation of the AIET. Whether or not the AIER has an introductory website and whether or not that website explains the internal mechanisms and techniques of the AIER determines the transparency of this component.

The transparency layer will help teachers to understand the possibilities, limitations and risks of that AIER in education. The third group is the Technology and Algorithm one. This class is to clarify the approaches the AIER used to reach its purpose and to help teachers understand how it works. The core questions to ask in these layers are as follows:

<table><tr><td rowspan="3">Technology</td><td>What functions are supported by AI technologies?</td></tr><tr><td>What is the AI technology for each of these functions?</td></tr><tr><td>What is the direct result of each of this AI technology?</td></tr></table>

Technology layer describes the AI technologies used in the AIER. Considering that an AIER may be an educational platform, a standalone or online application, etc., several different technologies may be used in a single resource. The purpose of this layer is to clarify whether AI-based technologies are actually being used and to introduce teachers to what kind of AI technologies are being used.

The technology could be one of the following list:

* Automatic generation of educational content (Courses, texts and so on).

* Improved educational content

* AI assistance for teachers to provide personalized instruction for each student.

* Fast feedback to students.

* Assistance in monitoring students.

* Automatic assesses students' learning behaviour and learning routes.

* Automatic records the student's learning process.

<figure><img src="image_30.png" /></figure>
<figure><img src="image_31.png" /></figure>

* Targeted improvement based on learning analysis.

* Chatbot between teachers, students, parents and relevant groups.

* Other technology for educational tasks.

<table><tr><td rowspan="2">Algorithm</td><td>For each of the above AI technologies, what are the algorithms?</td></tr><tr><td>To which family of approaches do the algorithms belong?</td></tr></table>

Algorithm layer describes the algorithms used in the AIER. Compared to the previous layer, this layer focuses more on the scientific level than on the technical level. We will declare the family of the algorithms from three common group: Knowledge-based Systems, Machine Learning and Deep Learning. Note that we are aware of the debate about whether deep learning is machine learning or not. However, we have separated them here because it is easier for teachers to understand. For each group, the algorithm could be selected from the following list:

* Knowledge-based Systems: Rule-based systems (or expert systems), Ontology, Semantic networks and so on.

* Machine Learning: Clustering, Approximate possibility, Regression Analysis, Representation and Dimensionality reduction, Active learning, Decision Trees, and so on.

* Deep Learning: Convolutional Networks (CNNs), Long Short-Term Memory Networks (LSTMs), Recurrent Neural Network (RNN), Generative Adversarial Networks (GANs).

**Please note that we will only identify the algorithms used in this layer and briefly introduce the algorithms, without comparing and evaluating the algorithms used.**

Overall, this 3-class 7-layer template can describe AIERs by introducing and charity the basic features of AIERs and the dimensions of interest to those who use AIERs in education (e.g. teachers, etc.), more importantly, this template can be targeted to increase user trust.

It is worth noting that the primary use of this template is for AI4T projects, designed to be teacher-facing. As such, we have discarded some elements that are less important or redundant for an educational environment, such as the operating environment, operation and maintenance staff, support for requirements, etc. However, following the methodology used to propose this template, it could easily be extended to a template for describing AIERs in other situations or other AI resources for use in a wider range of situations.

<figure><img src="image_32.png" /></figure>
## 3. Methodology for the questionnaire

### How should we design the questionnaire.

In order to fill in the template mentioned above, a questionnaire was designed for the providers of AIERs. Note that, though the questionnaire's purpose is to create examples of template for teachers and pedagogical experts, the questionnaire is fulfilled by the designers to investigate the characteristics of the AIER. Therefore, the design and order of the questions in the questionnaire are from the view of the AIER designer, rather than following the order of our proposed template.

In the questionnaire, we used colors to express the correspondence between the questionnaire questions and the various layers of the templates. After obtaining the completed questionnaire, we will fill in the examples of the template according to the correspondence.

**Note that, the options and full details of the questionnaire is not shown in this report, you can test it by the link:**

https://sondages.inria.fr/index.php/984292?lang=en

The questions in the questionnaire were designed into five groups, ranging from easy to answer to more involved details:

<table><tr><td rowspan="4">1.What is this resource?</td><td>Name of the resource</td></tr><tr><td>Short description of the resource</td></tr><tr><td>Information of the publisher of this resource</td></tr><tr><td>Is there a public information website about the resource?<br>If so, what is the URL?</td></tr></table>

The first group of questions was on basic AIER information, which asked for a short description in order to cross-validate additional information from the responses. It also included a request for a public information website to support the Transparency layer.

<table><tr><td rowspan="2">2.What is this resource used for?</td><td>Who is this resource primarily intended for?</td></tr><tr><td>What is the purpose of the resource for the user?</td></tr></table>

<figure><img src="image_33.png" /></figure>
<table><tr><td></td><td>What is the work of the teachers supported by this resource?</td></tr></table>

The second group of questions is about the use of AIER and relates to the use of layers. We offer several options for these questions as well. The main users can be students, teachers, educational institutions or others. For the tasks of teachers supported by AIER, we give options for four main areas of teacher work: developing and issuing educational content, delivering personalized instruction to each student, assessing and recording the learning process of students, supervising students' learning environment.

<table><tr><td rowspan="5">3.How does this resource achieve its purpose?</td><td>What are the functions of the AI technologies used in this resource?</td></tr><tr><td>For each of these functions, what is the AI technology used?</td></tr><tr><td>What are the algorithms used in these AI technology? (There is a list we used for options in the following table)</td></tr><tr><td>Is information about the algorithms used published on the public website of the resource?</td></tr><tr><td>Are there any scientific publications on the algorithms implemented? If so, what are the related published papers or documents?</td></tr><tr><td></td><td>Is open source AI software or resources used in this resource? If so, what is it?</td></tr></table>

The third group of questions is about AIER and the functionality of the technology. Note that, when collecting information on the Technology layer, we only discussed the technical information for exactly each function, not the usage of the entire tool or application. For each function, we asked respondents to indicate whether AI technology was used. We also asked for details of the algorithm of the AI technology, not only to gather information on the Algorithm layer, but also to validate the information on the Transparency layer. We gave the following list of algorithms for the options in this set of questions:
<table><thead><tr><th colspan="2">List of algorithms</th></tr></thead><tbody><tr><td rowspan="3">Knowledge-based Systems</td><td>Rule-based systems (or expert systems)</td></tr><tr><td>Ontology</td></tr><tr><td>Semantic networks</td></tr><tr><td rowspan="7">Machine Learning</td><td>Clustering(K-Means, Spectral Clustering, Hierarchical Clustering, etc.)</td></tr><tr><td>Approximate possibility(Markov Process, Markov Random Fields, Expectation Maximisation, etc.)</td></tr><tr><td>Regression Analysis(Linear Regression, Logistic Regression, etc.)</td></tr><tr><td>Representation and Dimensionality reduction(Metric Learning, Principal Component Analysis (PCA), Principal Component Analysis (PCA), etc.)</td></tr><tr><td>Active learning(Neural Network (ANN), Support Vector Machine (SVM), etc.)</td></tr><tr><td>Decision Trees, Random Forests, etc.</td></tr><tr><td>Reinforcement Learning (Q-Learning, SARSA, Armed Bandit, etc.)</td></tr><tr><td rowspan="4">Deep Learning</td><td>Convolutional Networks (CNNs)</td></tr><tr><td>Long Short Term Memory Networks (LSTMs)</td></tr><tr><td>Recurrent Neural Network (RNN)</td></tr><tr><td>Generative Adversarial Networks (GANs)</td></tr><tr><td rowspan="3">4.What does the resource require as input?</td><td>For the main AI functions of the resource you answered earlier, what type of data does the resource need to collect?</td></tr><tr><td>Besides the above data to support AI functions, does the need to collect additional personal data for creating login accounts or other non-AI functions? If so, what is the additional personal data collected?</td></tr><tr><td>Is data generated during interaction with the resource (assess log, click log or other trace, etc) collected? If so, what is generated data collected and for what purpose?</td></tr></tbody></table>

<figure><img src="image_34.png" /></figure>
<table><tr><td></td><td>Is any data collected other than personal data (environment in which the resource is running, other parameters to be entered, etc.)? If so, what are the additional data collected?</td></tr><tr><td></td><td>Is the personal data anonymous? If so, how does it work?</td></tr><tr><td></td><td>What is the method of secure data processing used?</td></tr><tr><td></td><td>How the resource collects consent under local regulation?</td></tr><tr><td></td><td>Can the personal data used in this resource be audited by third parties? If so, what personal data can be audited and how can it be accessed?</td></tr><tr><td></td><td>Can all of the above information on data collection be found on the relevant website of the resource?</td></tr></table>

The fourth group of questions is about the input of resources, which relates to information on the Personal data layer and the Data security layer. This group of questions asks for details of the data used by the AIER and cross-validates through a number of questions whether additional data is being collected. Information about the Data security layer is facilitated by asking about how this data is processed.

<table><tr><td rowspan="7">5. What is this resource output?</td><td>What is the directly result of the functions using AI?</td></tr><tr><td>What is the final result of the resource?</td></tr><tr><td>What is the presentation of this final result?</td></tr><tr><td>For these results, who has direct access?</td></tr><tr><td>Can teachers modify or change these results? If so, how can teachers affect these results?</td></tr><tr><td>What type of end result does this belong to?</td></tr><tr><td>How does the work of the teacher interact with this output?</td></tr><tr><td></td></tr></table>
<figure><img src="image_35.png" /></figure>

<figure><img src="image_36.png" /></figure>

The fifth group of questions is about the output of the AIER. The result of each AI function and the whole AIER and the right of assess, modify and rewrite are related to the information on Decision Level layer and Transparency layer. From the results on the AIER and the relationship between teachers and these results, we can verify the choices teachers can make when using this AIER and the extent to which the company publishing this AIER are open to teachers.

<figure><img src="image_37.png" /></figure>
<figure><img src="image_38.png" /></figure>

## 4. Leaflet and template

### Key information directly shown to teacher.

In this section, we will describe the leaflet for use directly to teachers. Note that the template we designed in the previous section, which is mainly for information collected from AI companies, while it is useful for AI experts and educational researchers with relevant knowledge, can be difficult for teachers to understand. It was therefore necessary to turn the template into a leaflet that was short enough to be easy to read, yet complex enough to include key information.

Here is an example:

<table><tr><td rowspan="2">Basic Information</td><td>Name of the resource:</td><td></td><td></td></tr><tr><td>Short description:</td><td></td><td></td></tr><tr><td rowspan="5">Usage</td><td>• For teacher</td><td>• For student</td><td>• For Educational Institutions</td></tr><tr><td>• Educational Generation</td><td>• Content</td><td>• Learning help</td><td>• Helping learning analysis</td></tr><tr><td>• Struggling identification</td><td>• student</td><td>• Skill develop</td><td>• Communication help</td></tr><tr><td>•</td><td></td><td>•</td><td>•</td></tr><tr><td>•</td><td></td><td></td><td>•</td></tr></table>

<figure><img src="image_39.png" /></figure>
<figure><img src="image_40.png" /></figure>

<table><tr><td></td><td>Classroom management help</td><td>Personal guide</td><td>Student conduct record</td></tr><tr><td></td><td>Other usage:</td><td></td><td></td></tr><tr><td></td><td>Main supported educational tasks:</td><td></td><td></td></tr><tr><td>Decision level</td><td>Production type of the resource:</td><td>Decision</td><td></td></tr><tr><td></td><td></td><td>Recommendation</td><td></td></tr><tr><td></td><td></td><td>Propose</td><td></td></tr><tr><td></td><td></td><td>Service</td><td></td></tr><tr><td></td><td>Accessible:</td><td>Student</td><td></td></tr><tr><td></td><td></td><td>Teacher</td><td></td></tr></table>

<figure><img src="image_41.png" /></figure>
<figure><img src="image_42.png" /></figure>

<table><tr><td></td><td><ul><li></li><li>User</li><li></li><li>Other: ____</li><li></li></ul></td></tr><tr><td>Teacher can:</td><td><ul><li>View</li><li></li><li></li><li>Change</li><li></li><li></li><li>Rewrite</li><li></li></ul></td></tr><tr><td>Responsible person:</td><td><ul><li>AI</li><li></li><li></li><li>User</li><li></li><li></li><li>Teacher</li></ul></td></tr></table>

<figure><img src="image_43.png" /></figure>
<table><tr><td></td><td></td><td>•<br>•<br>Other: ____<br>•</td></tr><tr><td></td><td>Teacher involves in the production:</td><td>•<br>Yes<br>•<br>No<br>•</td></tr><tr><td></td><td>If so, how:</td><td></td></tr><tr><td rowspan="5">Personal data</td><td>General type of personal data:</td><td>Collected and used for AI: •<br>Collected and used for other service: •</td></tr><tr><td>Name (First name, family name, etc.)</td><td>•</td></tr><tr><td>Personal physical information<br>(Gender, date of birth, age, etc.)</td><td>•</td></tr><tr><td>Contact information<br>(Phone number, email address, etc.)</td><td>•</td></tr><tr><td>Social information<br>(Cultural or social identity, etc.)</td><td>•</td></tr></table>

<figure><img src="image_44.png" /></figure>
<figure><img src="image_45.png" /></figure>

<table><tr><td></td><td>Internet information<br/>(IP number, cookie, credit number, etc.)</td><td>•</td><td>•</td></tr><tr><td></td><td>Data generated during interaction with resources (access logs, operation logs, etc.)</td><td>•</td><td>•</td></tr><tr><td></td><td>Other: __________</td><td>•</td><td>•</td></tr><tr><td rowspan="6">Data security</td><td>Encryption level:</td><td colspan="2">•<br/>Saved<br/>•<br/>•<br/>Anonymized<br/>•<br/>•<br/>Encrypted<br/>•</td></tr><tr><td>Encryption method</td><td colspan="2">•</td></tr><tr><td>Third party access:</td><td colspan="2">Yes<br/>•<br/>•<br/>No<br/>•</td></tr></table>

<figure><img src="image_46.png" /></figure>
<figure><img src="image_47.png" /></figure>

<table><tr><td></td><td>If so, who can visit:</td><td></td></tr><tr><td></td><td>Local regulations for consent:</td><td></td></tr><tr><td rowspan="7">Transparency</td><td>Outcome of this resource:</td><td></td></tr><tr><td>Presented as:</td><td></td></tr><tr><td>Audible software:</td><td><ul><li>Yes</li><li></li><li></li><li>No</li></ul></td></tr><tr><td>Website for explanation:</td><td></td></tr><tr><td>Related published papers or documents:</td><td></td></tr><tr><td>Name of the AI technology:</td><td></td></tr><tr><td>Function of this AI:</td><td></td></tr><tr><td rowspan="3">Algorithm</td><td>Name of the algorithm:</td><td></td></tr><tr><td>Knowledge-based Systems</td><td>Machine Learning</td><td>Deep Learning</td></tr><tr><td></td><td></td><td></td></tr></table>

<figure><img src="image_48.png" /></figure>
<figure><img src="image_49.png" /></figure>

<table><tr><td></td><td>Classification</td><td>Prediction</td><td>Regression</td></tr><tr><td></td><td>•</td><td>•</td><td>•</td></tr><tr><td></td><td>•</td><td>•</td><td>•</td></tr><tr><td></td><td>Recommendation</td><td>Knowledge discovery</td><td>Clustering</td></tr><tr><td></td><td>•</td><td>•</td><td>•</td></tr><tr><td></td><td>•</td><td></td><td></td></tr><tr><td></td><td>Other:</td><td></td><td></td></tr><tr><td></td><td>•</td><td></td><td></td></tr></table>

<figure><img src="image_50.png" /></figure>
## 5. Example of the template

### How should we use the template?

Here are some examples we get:

<table><tr><td>Basic Information</td><td>Name of the resource<br/>Short description</td><td>Duolingo for Schools<br/>Duolingo for Schools is a dashboard within a teacher's Duolingo account that allows teachers to create classrooms and assignments and track student activity to improve teacher effectiveness and student learning.</td></tr><tr><td rowspan="3">Usage</td><td>Who is this resource primarily intended for?</td><td>Teacher</td></tr><tr><td>What is the intended use of this resource?</td><td>Helping teacher create classrooms and assignments and track student activity on Duolingo</td></tr><tr><td>What area of teachers' work does this resource support?</td><td>Assessing and recording the learning process of students</td></tr><tr><td rowspan="4">Decision level</td><td>What is the type of output of the AI in the resource? Is it a decision, a recommendation, a proposal or a service?</td><td>Service</td></tr><tr><td>What results does the AI give in the resource?</td><td>Adapt learning pathway of students or of group of student</td></tr><tr><td>Who can see the result of this resource?</td><td>Teacher</td></tr><tr><td>Can teachers access, modify, rewrite or otherwise influence the results given by the AI?</td><td>Yes, they could access the result and modify part of the elements on the dashboard</td></tr></table>
<table><tr><td></td><td>Are teachers involved in the production of the result? If so, how do they do this?</td><td>The teacher decides how to task or assess the students and can present two types of tasks to the learners, the first being to collect experience points (or XP) and the second being to level up a skill.</td></tr><tr><td></td><td>Who is responsible for the result?</td><td>Teacher</td></tr><tr><td rowspan="3">Personal data</td><td>What personal data is collected for the needs of the AI technology in the resource?</td><td>Nothing</td></tr><tr><td>What personal data is collected for other functions in addition to the AI technology in the resource?</td><td>Teachers' emails are collected for creating accounts</td></tr><tr><td>Is data generated during interaction with the resource collected? If so, for what purpose?</td><td>Students' activities are collected for generating the elements on the dashboard</td></tr><tr><td rowspan="3">Data security</td><td>Is the personal data anonymous? If so, how does it work?</td><td>Anonymized<br>Allows the creation of profiles without name, surname or email addresses;<br>Teachers can also create accounts for their students without using name or email address.</td></tr><tr><td>How the resource collects consent under local regulation?</td><td>No consent procedure</td></tr><tr><td>Can the personal data used in this resource be audited by third parties? If so, what personal data can be audited and how can it be accessed?</td><td>No</td></tr></table>
<figure><img src="image_51.png" /></figure>

<table><tr><td></td><td>What are the possibilities for outsiders to audit the data in this resource?</td><td>No</td></tr><tr><td rowspan="5">Transparency</td><td>What is the outcome or end service provided by this resource?</td><td>A report of student's behavior.</td></tr><tr><td>In what way is this result presented to the user?</td><td>A dashboard right inside of a teacher's Duolingo account.</td></tr><tr><td>Is there a website or other publicly available way to explain how this resource works? If so, what is it?</td><td>https://research.duolingo.com/</td></tr><tr><td>For the AI techniques used in the resources, are there any relevant published papers or documents on the above website or other accessible sources? If so, what is it?</td><td>Incidentally many scientific articles are published by the Duolingo teams in scientific publication to explain how some elements of the model work : https://research.duolingo.com/ https://research.duolingo.com/papers/yancey.kdd20.pdf https://sharedtask.duolingo.com/papers/chen.slam18.pdf</td></tr><tr><td>Is the software used in this resource audible? If so, how does it work?</td><td>Unknown</td></tr><tr><td rowspan="3">Technology</td><td>What functions are supported by AI technologies?</td><td>Providing analytical reports on students managed under teacher accounts</td></tr><tr><td>What is the AI technology for each of these functions?</td><td>Learning analysis of student information</td></tr><tr><td>What is the directly result of each of this AI technology?</td><td>Analysis report</td></tr></table>

<figure><img src="image_52.png" /></figure>
<figure><img src="image_53.png" /></figure>

<table><tr><td rowspan="2">Algorithm</td><td>For each of the above AI technologies, what are the algorithms?</td><td>Sleeping recovering Bandit; Bayesian Knowledge training</td></tr><tr><td>To which family of approaches do the algorithms belong?</td><td>Machine learning</td></tr></table>

<figure><img src="image_54.png" /></figure>
<figure><img src="image_55.png" /></figure>

<table><tr><td>Basic Information</td><td>Name of the resource<br/>Short description</td><td>Kwyk<br/>Kwyk is a training and evaluation website in mathematics and physics. Teachers have access to our exercise database to create their homeworks. These homeworks are self-corrected and self-graded.</td></tr><tr><td rowspan="3">Usage</td><td>Who is this resource primarily intended for?</td><td>Teacher</td></tr><tr><td>What is the intended use of this resource?</td><td>Create homework assignments with automatic corrections and records</td></tr><tr><td>What area of teachers' work does this resource support?</td><td>Assessing and recording the learning process of students</td></tr><tr><td rowspan="5">Decision level</td><td>What is the type of output of the AI in the resource? Is it a decision, a recommendation, a proposal or a service?</td><td>Service</td></tr><tr><td>What results does the AI give in the resource?</td><td>Prediction of students' success in a given exercise</td></tr><tr><td>Who can see the result of this resource?</td><td>Teacher</td></tr><tr><td>Can teachers access, modify, rewrite or otherwise influence the results given by the AI?</td><td>Yes, they could access the result.</td></tr><tr><td>Are teachers involved in the production of the result? If so, how do they do this?</td><td>They could use them for future education plan</td></tr><tr><td></td><td>Who is responsible for the result?</td><td>No answer</td></tr></table>

<figure><img src="image_56.png" /></figure>
<table><tr><td rowspan="3">Personal data</td><td>What personal data is collected for the needs of the AI technology in the resource?</td><td>Internet information include cookies</td></tr><tr><td>What personal data is collected for other functions in addition to the AI technology in the resource?</td><td>Teachers' emails are collected for creating accounts</td></tr><tr><td>Is data generated during interaction with the resource collected? If so, for what purpose?</td><td>Students' answers are collected for the prediction</td></tr><tr><td rowspan="4">Data security</td><td>Is the personal data anonymous? If so, how does it work?</td><td>Teacher data is unencrypted but not public, and students can log in anonymously.</td></tr><tr><td>How the resource collects consent under local regulation?</td><td>By validating the general conditions of use</td></tr><tr><td>Can the personal data used in this resource be audited by third parties? If so, what personal data can be audited and how can it be accessed?</td><td>No</td></tr><tr><td>What are the possibilities for outsiders to audit the data in this resource?</td><td>No</td></tr><tr><td rowspan="2">Transparency</td><td>What is the outcome or end service provided by this resource?</td><td>Kwyk provides generated assignments based on the teacher's choices and provides the teacher with statistical information on the completion of assignments and related predictions based on student responses.</td></tr><tr><td>In what way is this result presented to the user?</td><td>Kwyk provides dashboards showing results to teachers and students answering questions</td></tr></table>
<figure><img src="image_57.png" /></figure>

<table><tr><td></td><td>Is there a website or other publicly available way to explain how this resource works? If so, what is it?</td><td>https://www.kwyk.fr/questions/apropos/</td></tr><tr><td></td><td>For the AI techniques used in the resources, are there any relevant published papers or documents on the above website or other accessible sources? If so, what is it?</td><td>No</td></tr><tr><td></td><td>Is the software used in this resource audible? If so, how does it work?</td><td>No</td></tr><tr><td rowspan="3">Technology</td><td>What functions are supported by AI technologies?</td><td>Predicting student success on specific exercises based on the exercises they answer</td></tr><tr><td>What is the AI technology for each of these functions?</td><td>Predictions of student performance based on learning data</td></tr><tr><td>What is the direct result of each of this AI technology?</td><td>The prediction result</td></tr><tr><td rowspan="2">Algorithm</td><td>For each of the above AI technologies, what are the algorithms?</td><td>Boosting</td></tr><tr><td>To which family of approaches do the algorithms belong?</td><td>Classification</td></tr></table>

<table><tr><td rowspan="2">Basic Information</td><td>Name of the resource</td><td>Quiz Wizard</td></tr><tr><td>Short description</td><td>Quiz Wizard helps teachers and trainers generate quiz with AI</td></tr><tr><td>Usage</td><td>Who is this resource primarily intended for?</td><td>Teacher</td></tr></table>

<figure><img src="image_58.png" /></figure>
<figure><img src="image_59.png" /></figure>

<table><tr><td></td><td>What is the intended use of this resource?</td><td>Automatic educational materials generation</td></tr><tr><td></td><td>What area of teachers' work does this resource support?</td><td>Developing and issuing educational content</td></tr><tr><td rowspan="5">Decision level</td><td>What is the type of output of the AI in the resource? Is it a decision, a recommendation, a proposal or a service?</td><td>Proposition</td></tr><tr><td>What results does the AI give in the resource?</td><td>The content (the MCQs or the flashcards)</td></tr><tr><td>Who can see the result of this resource?</td><td>The user of the resource</td></tr><tr><td>Can teachers access, modify, rewrite or otherwise influence the results given by the AI?</td><td>Yes teacher could edit the result.</td></tr><tr><td>Are teachers involved in the production of the result? If so, how do they do this?</td><td>Teacher could edit the generate content</td></tr><tr><td></td><td>Who is responsible for the result?</td><td>No answer</td></tr><tr><td rowspan="3">Personal data</td><td>What personal data is collected for the needs of the AI technology in the resource?</td><td>Context is used to create the content. Context can be a topic, the text content of a document, or a link to a website for instance.</td></tr><tr><td>What personal data is collected for other functions in addition to the AI technology in the resource?</td><td>Name, email address, and organization name</td></tr><tr><td>Is data generated during interaction with the resource collected? If so, for what purpose?</td><td>No</td></tr></table>

<figure><img src="image_60.png" /></figure>
<figure><img src="image_61.png" /></figure>

<table><tr><td rowspan="4">Data security</td><td>Is the personal data anonymous? If so, how does it work?</td><td>Anonymized.<br/>No personal data sent to the LLM, so anonymous. But personal data used for account management.</td></tr><tr><td>How the resource collects consent under local regulation?</td><td>Consent is asked when creating an account.</td></tr><tr><td>Can the personal data used in this resource be audited by third parties? If so, what personal data can be audited and how can it be accessed?</td><td>No</td></tr><tr><td>What are the possibilities for outsiders to audit the data in this resource?</td><td>No</td></tr><tr><td rowspan="5">Transparency</td><td>What is the outcome or end service provided by this resource?</td><td>MCQs and flashcard</td></tr><tr><td>In what way is this result presented to the user?</td><td>Textual output. Different export formats (Wooclap, Wooflash, Moodle XML, Word and Excel)</td></tr><tr><td>Is there a website or other publicly available way to explain how this resource works? If so, what is it?</td><td>https://www.getquizwizard.com/</td></tr><tr><td>For the AI techniques used in the resources, are there any relevant published papers or documents on the above website or other accessible sources? If so, what is it?</td><td>https://openai.com/research</td></tr><tr><td>Is the software used in this resource audible? If so, how does it work?</td><td>No</td></tr></table>

<figure><img src="image_62.png" /></figure>
<figure><img src="image_63.png" /></figure>

<table><tr><td rowspan="3">Technology</td><td>What functions are supported by AI technologies?</td><td>generate MCQs or flashcards, from a topic, a document (PDF, Word, PPT etc), a link to a website, a text, or an audio/video file</td></tr><tr><td>What is the AI technology for each of these functions?</td><td>ChatGPT's API</td></tr><tr><td>What is the direct result of each of this AI technology?</td><td>The content (the MCQs or the flashcards)</td></tr><tr><td rowspan="2">Algorithm</td><td>For each of the above AI technologies, what are the algorithms?</td><td>LLM with transformers</td></tr><tr><td>To which family of approaches do the algorithms belong?</td><td>Deep learning</td></tr></table>

<figure><img src="image_64.png" /></figure>
<figure><img src="image_65.png" /></figure>

## 6. Reference

[1] European Schoolnet. "Artificial Intelligence Role in K12 Education: Agile Collection of Information", 2021, Brussels, Belgium.

[2] Amit Gautam "Role Of Artificial Intelligence In Shaping The Future Workforce", July 7, 2019, eLearning Industry, https://elearningindustry.com

[3] Karine Perset, Dewey Murdick, Jack Clark, Marko Grobelink "A first look at the OECD's Framework for the Classification of AI Systems, designed to give policymakers clarity", November 24, 2020, https://oecd.ai/en/

[4] Tahiru, Fati. "AI in education: A systematic literature review." Journal of Cases on Information Technology (JCIT) 23.1 (2021): 1-20.

[5] Khosravi, Hassan, et al. "Explainable artificial intelligence in education." Computers and Education: Artificial Intelligence 3 (2022): 100074.

[6] Sarah Hunt. "How AI is Being Used in Education", October 10, 2021, https://www.datamation.com/

[7] Eyupoglu C, Aydin M A, Zaim A H, et al. An efficient big data anonymization algorithm based on chaos and perturbation techniques[J]. Entropy, 2018, 20(5): 373.

[8] Ouyang F, Jiao P. Artificial intelligence in education: The three paradigms[J]. Computers and Education: Artificial Intelligence, 2021, 2: 100020.

[9] Chen L, Chen P, Lin Z. Artificial intelligence in education: A review[J]. Ieee Access, 2020, 8: 75264-75278.

<figure><img src="image_66.png" /></figure>