# Impact of Item and Attribute Design on Intelligent Tutoring Systems' Performance

Ekaterina Goliakova

<figure><img src="image_1.png" /></figure>

To cite this version:

Ekaterina Goliakova. Impact of Item and Attribute Design on Intelligent Tutoring Systems' Performance. Artificial Intelligence [cs.AI]. 2024. hal-05313626

HAL Id: hal-05313626

https://inria.hal.science/hal-05313626v1

Submitted on 14 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
UNIVERSITÉ DE LORRAINE

MASTER THESIS

# Impact of Item and Attribute Design on Intelligent Tutoring Systems' Performance

Author:
Ekaterina GOLIAKOVA

Supervisor:
Marc DINH

Co-supervisor:
Dr. Benedict O'DONNELL

Reviewer:
Prof. Miguel COUCEIRO

A thesis submitted in fulfillment of the requirements
for the degree of Master of Science in Natural Language Processing
in

Naimrod, Paris
IDMC, Université de Lorraine

IDMC
Institut des
sciences du Digital
Management & Cognition

UNIVERSITÉ
DE LORRAINE

<figure><img src="image_3.png" /></figure>
<figure><img src="image_4.png" /></figure>
**Declaration of Authorship**

I, Ekaterina GOLIAKOVA, declare that this thesis titled, "Impact of Item and Attribute Design on Intelligent Tutoring Systems' Performance" and the work presented in it are my own. I confirm that:

* This work was done wholly or mainly while in candidature for a research degree at this University.

* Where any part of this thesis has previously been submitted for a degree or any other qualification at this University or any other institution, this has been clearly stated.

* Where I have consulted the published work of others, this is always clearly attributed.

* Where I have quoted from the work of others, the source is always given. With the exception of such quotations, this thesis is entirely my own work.

* I have acknowledged all main sources of help.

* Where the thesis is based on work done by myself jointly with others, I have made clear exactly what was done by others and what I have contributed myself.

Signed:

Date:
<figure><img src="image_5.png" /></figure>
<figure><img src="image_6.png" /></figure>
UNIVERSITÉ DE LORRAINE

Abstract

IDMC

Université de Lorraine

Master of Science in Natural Language Processing

Impact of Item and Attribute Design on Intelligent Tutoring Systems'
Performance

by Ekaterina GOLIAKOVA

Intelligent Tutoring Systems (ITS) are slowly becoming an integral part of modern education, and in particular, language learning. However, the design of such systems is plagued by ambiguity: what skills are supposed to be taught, what is the allowed level of mistakes, and what questions are most representative of students' mastery of skills? Despite the prominence of these questions, the research on the exact impact of these parameters is still lacking. In this study, we attempt to bridge this gap by comparing different definitions of attributes and the corresponding performance of the system.
<figure><img src="image_7.png" /></figure>
# Contents

<table><tr><td>Declaration of Authorship</td><td>iii</td></tr><tr><td>Abstract</td><td>vii</td></tr><tr><td>1 Introduction</td><td>1</td></tr><tr><td>1.1 Project Motivation</td><td>1</td></tr><tr><td>1.2 Project Objective</td><td>2</td></tr><tr><td>2 Background</td><td>5</td></tr><tr><td>2.1 Introduction to ITS</td><td>5</td></tr><tr><td>2.2 Item and Competency Design</td><td>6</td></tr><tr><td>2.3 Knowledge Model Design</td><td>8</td></tr><tr><td>2.3.1 Knowledge Space Theory</td><td>9</td></tr><tr><td>2.3.2 KST and Formal Concept Analysis</td><td>10</td></tr><tr><td>2.4 Student Modelling</td><td>12</td></tr><tr><td>2.5 Other Methodologies</td><td>14</td></tr><tr><td>Bayesian Knowledge Tracing</td><td>14</td></tr><tr><td>Item Response Theory</td><td>14</td></tr><tr><td>3 Implementation</td><td>15</td></tr><tr><td>3.1 Approach Outline</td><td>15</td></tr><tr><td>3.2 Dataset</td><td>15</td></tr><tr><td>3.2.1 Data Labeling</td><td>16</td></tr><tr><td>Skill Map</td><td>16</td></tr><tr><td>Difficulty Labelling</td><td>16</td></tr><tr><td>Complexity Labelling</td><td>17</td></tr><tr><td>3.2.2 Student Profiles</td><td>18</td></tr><tr><td>3.3 Knowledge Space Creation</td><td>20</td></tr><tr><td>3.3.1 Converting Answers to Formal Context</td><td>20</td></tr><tr><td>3.3.2 Mining Frequent Itemsets</td><td>20</td></tr><tr><td>3.3.3 Mining Surmise Relations</td><td>21</td></tr><tr><td>3.4 Building a Learning Path</td><td>21</td></tr><tr><td>3.4.1 Learning Edge Weights</td><td>22</td></tr><tr><td>3.4.2 Learning Path from Learned Weights</td><td>23</td></tr><tr><td>3.4.3 Building a Baseline Learning Path</td><td>24</td></tr><tr><td>3.4.4 Path Post-Processing</td><td>25</td></tr><tr><td>3.5 Evaluation</td><td>25</td></tr><tr><td>3.5.1 Difficulty Correlation</td><td>26</td></tr><tr><td>3.5.2 Complexity Correlation</td><td>26</td></tr><tr><td>3.5.3 Reproducibility</td><td>26</td></tr></table>
<table><tr><td>4 Experiments</td><td>27</td></tr><tr><td>4.1 Granularity of Attributes</td><td>27</td></tr><tr><td>4.2 Comparing Mastery Thresholds</td><td>30</td></tr><tr><td>4.3 Comparing Difficulty of Questions in Attributes</td><td>32</td></tr><tr><td>4.4 Comparing with the Baseline</td><td>34</td></tr><tr><td>5 Discussion</td><td>37</td></tr><tr><td>6 Conclusion</td><td>39</td></tr><tr><td>6.1 Limitations</td><td>39</td></tr><tr><td>6.2 Future Work</td><td>40</td></tr><tr><td>6.2.1 Granularity of Attributes</td><td>40</td></tr><tr><td>6.2.2 Skill Map and Question Design</td><td>41</td></tr><tr><td>6.2.3 Unconscious Biases in the ITS Design</td><td>41</td></tr><tr><td>6.2.4 Impact of Student Profiles</td><td>41</td></tr><tr><td>6.3 Main Findings</td><td>41</td></tr><tr><td>A Attributes</td><td>43</td></tr><tr><td>A.1 Attribute Distribution</td><td>43</td></tr><tr><td>A.1.1 Coarse Attributes</td><td>43</td></tr><tr><td>A.1.2 Finer Attributes</td><td>45</td></tr><tr><td>A.2 Question Examples</td><td>47</td></tr><tr><td>A.2.1 CEFR A1-A2 Attributes</td><td>47</td></tr><tr><td>A.2.2 CEFR B1.1 Attributes</td><td>48</td></tr><tr><td>A.2.3 CEFR B1.2-B2.1 Attributes</td><td>49</td></tr><tr><td>B Learning Paths</td><td>51</td></tr><tr><td>B.1 All Students</td><td>51</td></tr><tr><td>B.2 Beginners</td><td>52</td></tr><tr><td>B.3 Intermediate</td><td>53</td></tr><tr><td>B.4 Advanced</td><td>54</td></tr><tr><td>Bibliography</td><td>55</td></tr></table>
List of Figures

<table>
  <tbody>
    <tr>
      <td>1.1</td>
      <td>A question example from a sample TOEIC exam (Lougheed, 2013).</td>
      <td>2</td>
    </tr>
    <tr>
      <td>2.1</td>
      <td>The trade-off between granularity of attributes and coverage of ITS (from Pelánek, 2020).</td>
      <td>7</td>
    </tr>
    <tr>
      <td>2.2</td>
      <td>A representation of a sample knowledge space designed using the knowledge domain from Table 2.3, taken from Ganter et al., 2017.</td>
      <td>9</td>
    </tr>
    <tr>
      <td>2.3</td>
      <td>created for the formal context in Table 2.4 and the corresponding knowledge space.</td>
      <td>13</td>
    </tr>
    <tr>
      <td>3.1</td>
      <td>The competencies of the GSE Teacher Toolkit associated with attribute Past Simple.</td>
      <td>19</td>
    </tr>
    <tr>
      <td>3.2</td>
      <td>Percetange of students corresponding to each student profile.</td>
      <td>19</td>
    </tr>
    <tr>
      <td>3.3</td>
      <td>An example of weight update during the 2-fold splitting with n repetitions.</td>
      <td>23</td>
    </tr>
    <tr>
      <td>4.1</td>
      <td>The percentage of questions for each CEFR level split by granularity of attributes.</td>
      <td>27</td>
    </tr>
    <tr>
      <td>4.2</td>
      <td>Metric results for coarse and finer attributes, aggregated (averaged) for different student groups. Figure 4.2d represents the aggregated (averaged) results of all 3 metrics, with tinted areas representing the standard deviation of results.</td>
      <td>29</td>
    </tr>
    <tr>
      <td>4.3</td>
      <td>Metric results for different threshold approaches, aggregated (averaged) for different student groups. Figure 4.3d represents the aggregated (averaged) results of all 3 metrics, with tinted areas representing the standard deviation of results.</td>
      <td>31</td>
    </tr>
    <tr>
      <td>4.4</td>
      <td>Aggregated (averaged) metric results for different thresholds, grouped by student profile.</td>
      <td>32</td>
    </tr>
    <tr>
      <td>4.5</td>
      <td>Metric results for different threshold approaches, aggregated (averaged) for different student groups. Figure 4.5d represents the aggregated (averaged) results of all 3 metrics, with tinted areas representing the standard deviation of results.</td>
      <td>33</td>
    </tr>
    <tr>
      <td>4.6</td>
      <td>Metric results for building a path using trained weights and using conditional probabilities, aggregated (averaged) for different student groups. Figure 4.6d represents the aggregated (averaged) results of all 3 metrics, with tinted areas representing the standard deviation of results.</td>
      <td>35</td>
    </tr>
    <tr>
      <td>6.1</td>
      <td>Dependency of runtime from the size of knowledge space. The regression line drawn at y = x/47.</td>
      <td>40</td>
    </tr>
    <tr>
      <td>A.1</td>
      <td>Number of questions associated with an attribute (coarse attributes).</td>
      <td>43</td>
    </tr>
    <tr>
      <td>A.2</td>
      <td>Success rates per coarse attribute for different student profiles (all, beginners, intermediate, advanced).</td>
      <td>44</td>
    </tr>
    <tr>
      <td>A.3</td>
      <td>Number of questions associated with an attribute (finer attributes).</td>
      <td>45</td>
    </tr>
  </tbody>
</table>
<table><tr><td>A.4 Success rates per finer attribute for different student profiles (all, be-ginners, intermediate, advanced).</td><td>45</td></tr></table>
List of Tables

<table><tbody><tr><td>2.1</td><td>An example of attribute granularity from Pelánek, 2020.</td><td>7</td></tr><tr><td>2.2</td><td>An example of conjunctive and disjunctive skill mapping of TOEIC questions. Correct answers are bolded. TOEIC questions are taken from Lougheed, 2013.</td><td>8</td></tr><tr><td>2.3</td><td>An example of knowledge domain for the field of algebra, taken from Ganter et al., 2017</td><td>10</td></tr><tr><td>2.4</td><td>An example formal context for students and the questions from the knowledge domain. We consider that a student possesses an attribute (skill) if they mastered it.</td><td>11</td></tr><tr><td>2.5</td><td>FCA notions and the corresponding KST notions.</td><td>12</td></tr><tr><td>3.1</td><td>CEFR's self-assessment for reading skills in the English language.</td><td>18</td></tr><tr><td>3.2</td><td>Binary context S × G representing mastery of skills by students.</td><td>20</td></tr><tr><td>4.1</td><td>Standard deviations of the number of sub-competences, GSE score, and the number of questions per attribute, using coarse and finer attributes.</td><td>28</td></tr><tr><td>4.2</td><td>Average success rates per "Verb Agreement" attributes. The coarse attribute is shown before the bold line and the corresponding finer attributes can be found after the bold line.</td><td>28</td></tr><tr><td>4.3</td><td>P-values for t-test performed comparing metric results for all students using coarse- and finer-grained attributes. The p-value threshold is set to be p-value ≤ 0.05.</td><td>29</td></tr><tr><td>4.4</td><td>T-test results (p-value and t-statistic) from comparing metric results for different threshold approaches. The p-value threshold is set to be p-value ≤ 0.05.</td><td>31</td></tr><tr><td>4.5</td><td>T-test results (p-value and t-statistic) from comparing metric results for different question selection approaches. The p-value threshold is set to be p-value ≤ 0.05.</td><td>34</td></tr><tr><td>4.6</td><td>P-values for t-test performed comparing metric results for all students using learning path from trained weights and based on conditional probabilities. The p-value threshold is set to be p-value ≤ 0.05.</td><td>35</td></tr><tr><td>A.1</td><td>Coarse attributes, their corresponding average GSE scores, CEFR levels, and success rates (SR) for different groups of students.</td><td>44</td></tr><tr><td>A.2</td><td>Finer attributes, their corresponding average GSE scores, CEFR levels, and success rates (SR) for different groups of students.</td><td>46</td></tr><tr><td>A.3</td><td>Attributes s with CEFR level A1-A2.2 and an example of q ∈ Qall returned by the problem function p(s). Note that multiple skills can be required to solve a question.</td><td>47</td></tr><tr><td>A.4</td><td>Attributes s with CEFR level B1.1 and an example of q ∈ Qall returned by the problem function p(s). Note that multiple skills can be required to solve a question.</td><td>48</td></tr><tr><td>A.5</td><td>Attributes s with CEFR level B1.1-B2.1 and an example of q ∈ Qall returned by the problem function p(s). Note that multiple skills can be required to solve a question.</td><td>49</td></tr></tbody></table>
B.1 The proposed learning path for all students. NB: Several skills can be proposed at the same step of the learning path. 51

B.2 The proposed learning path for beginners. NB: Several skills can be proposed at the same step of the learning path. 52

B.3 The proposed learning path for intermediate students. NB: Several skills can be proposed at the same step of the learning path. Step 0 stands for knowledge mastered by all intermediate students. 53

B.4 The proposed learning path for advanced students. NB: Several skills can be proposed at the same step of the learning path. Step 0 stands for knowledge mastered by all advanced students. 54
List of Abbreviations

<table><tr><td>EFL</td><td>English as Foreign Language</td></tr><tr><td>ITS</td><td>Intelligent Tutoring System</td></tr><tr><td>KST</td><td>Knowledge Space Theory</td></tr><tr><td>FCA</td><td>Formal Concept Analysis</td></tr><tr><td>TOEIC</td><td>Test of English for International Communication</td></tr><tr><td>STEM</td><td>Science, technology, engineering, and mathematics</td></tr><tr><td>NLP</td><td>Natural Language Processing</td></tr><tr><td>LCM</td><td>Linear time Closed itemset Mining</td></tr><tr><td>BLIM</td><td>Basic Local Independence Model</td></tr><tr><td>LA</td><td>Learning Analytics</td></tr><tr><td>CEFR</td><td>Common European Framework of Reference for Languages</td></tr><tr><td>GSE</td><td>Global Scale of English</td></tr></table>
<figure><img src="image_8.png" /></figure>
# Chapter 1

## Introduction

The thesis is based primarily on experiments performed during the alternance work at Naimrod<fnref n="1" />. Naimrod is an IT startup, located in Paris, France, with the core personnel of Dr. Benedict O'Donnell, CEO, and Marc Dinh, CTO. The company mission is to make learning binge-worthy through adaptable and immersive content with a primary focus on foreign language learning and teaching ².

In this chapter, you will find a brief introduction to the company, its structure, and the objective set during the course of work. Chapter 2 will cover the existing work in the field, while Chapter 3 will consist of describing our own approach and how it was implemented and evaluated. Chapter 4 will cover the experiments performed in the scope of the project, and Chapter 5 will provide a brief of discussion of the achieved results. Finally, the conclusion and a summary of performed work will be provided in Chapter 6.

### 1.1 Project Motivation

In the modern realia, the importance of learning English can not be undermined. With the widespread of the Internet where around 50% of all content is published in English³, the interest in learning English as a foreign language (EFL) keeps rising, making it the most spoken second language globally (Matthiessen, 2015). Additionally, some student believe that knowing English could help them professionally (Getie, 2020).

If one wants to teach or learn a language in the current times, they might choose to use an Intelligent Tutoring System (ITS), such as Duolingo⁴, Memrise⁵, Busuu⁶, Babbel⁷, and others, due to their easy accessibility. ITS can be defined as computerized learning environments that combine methods from the cognitive sciences, learning sciences, computational linguistics, AI, mathematics, and other fields (Guo et al., 2021). As shown by Guo et al., 2021, the research in the field of ITS has been booming between the years 2000 and 2019, with ITSs dominating the field of language education for the past 2 decades (Liang et al., 2023).

However, despite the long history (with the first applications of ITS of this kind appearing as early as in the 70s (Nwana, 1990)) and the recent popularity, it is clear that STEM sciences are disproportionately represented among ITS research: out of

<footnote n="1">¹https://naimrod.com/
²https://naimrod.com/about/
³According to W3Techs https://w3techs.com/technologies/history_overview/content_language/ms/y.
⁴https://www.duolingo.com/
⁵https://www.memrise.com/
⁶https://www.busuu.com/
⁷https://fr.babbel.com/</footnote>
53 studies reviewed by Mousavinasab et al., 2021, only 4 were related to languages, with a heavier focus on STEM fields like computer science or math.

One of the explanations for such a phenomenon could be the fact that languages broadly are *ill-defined domains*, compared to a *well-defined domain* (e.g., mathematics, physics) (Woolf, 2010). Well-defined domains can be characterized with the following attributes (Woolf, 2010):

* The unambiguity of answers, which can be either correct or incorrect. On the contrary, language ambiguity is prevalent and has a great effect on a foreign language acquisition (Chapelle and Roberts, 1986; Lyster, 1998);

* Clear-cut domain models that incorporate difficulty and pre-requisites. Conversely, when speaking about language teaching, the difficulty of topics is not well-defined and can be greatly affected by factors like the native language of the speaker and socio-cultural context (Rautionaho, Deshors, and Meriläinen, 2018).

## 1.2 Project Objective

Broadly speaking, the objective of this project is to bridge the aforementioned gap in the ITS creation for language learning domain.

Firstly, to address the potential ambiguity, the study was narrowed down to focus on building an ITS to acquire competencies required to pass TOEIC Listening & Reading exam<fnref n="8" />. TOEIC is one of the standardized tests used to assess students’ level in EFL. The test consists of 200 questions: 100 in the listening and 100 in the reading section. In this work, we focused exclusively on the questions and answers for the reading section.

<figure><img src="image_9.png" /><figcaption>FIGURE 1.1: A question example from a sample TOEIC exam (Lougheed, 2013).</figcaption></figure>

As can be observed in Figure 1.1, the exam is used to test students’ knowledge of a singular competency (in the given example, the competency in question is the knowledge of Past Perfect tense). Due to its rigid and multiple-choice structure, it has been argued that the results in the TOEIC exam do not correlate well with proficiency in English (Nguyen and Gu, 2020; Nicholson, 2015). However, we argue that exactly this kind of question format allows to alleviate the ambiguity of answers that is associated with language learning, and thus makes the data more suitable for an ITS creation. Additionally to addressing the ambiguity, such domain narrowing down allows for a more clear-cut difficulties and prerequisites.

As mentioned above, another factor making the creation of an ITS for language learning challenging is the absence of clear-cut difficulties and pre-requisites. While the usage of the TOEIC domain addresses the pre-requisite part of this problem,

<footnote n="8">⁸https://www.ets.org/toeic/test-takers/about/listening-reading.html</footnote>
the perceived difficulties might still be demanding to assess. In order to solve this issue, we use data from a group of students of a relatively homogenous socio-cultural background: French-speaking university-level students, preparing for a TOEIC exam.

With this, the project objective could be achieved. However, it is worth remembering that an ITS is a very versatile tool (Mousavinasab et al., 2021; Nwana, 1990), that can be applied to a multitude of tasks. Therefore we must highlight, that the main application of an ITS envisioned by Naimrod is to use such a system to provide teachers with a sequence of topics to teach their students (or a *learning path*).

Having this purpose in mind and given the previous experience of Naimrod with ITS (Calò, 2021), it was decided to create a system using a fusion of Knowledge Space Theory (KST) (Doignon and Falmagne, 1985; Doignon and Falmagne, 2012; Düntsch and Gediga, 1995) and Formal Concept Analysis (FCA) (Ganter and Kuznetsov, 2001; Ganter and Wille, 2012; Ganter and Stumme, 2005).

To further guide our work, the following questions were posed regarding the project:

* Given responses to a language proficiency test, can we create such an ITS that would recommend an EFL teacher a learning path that is optimal for their students?

* How a learning path recommendation can be evaluated in an *offline* environment (without access to data about how the system is used and its effectiveness)?

* With the limited data available, can we improve the performance of such a system by just changing parameters while using the same data?
<figure><img src="image_10.png" /></figure>
# Chapter 2

## Background

In this chapter, we will cover the main notions<fnref n="1" /> concerning ITS, KST, and FCA. Section 2.1 will give a more detailed definition of ITS via its main components, namely as an item, a skill, a knowledge model, and a student model. Section 2.2 will present the challenges regarding appropriate item and skill selection. Section 2.3 will discuss how a knowledge model can be constructed using KST and FCA fusion, while Section 2.4 will present difficulties associated with modeling students’ behavior. Finally, Section 2.5 will provide a brief mention of other possible approaches to the ITS construction.

### 2.1 Introduction to ITS

While 20-30 years ago researchers were skeptical about the effectiveness of ITS applied to real-life education environments (Self, 1988; Cumming and McDougall, 2000, as cited by Desmarais and Baker, 2012), a recent meta-review about the role of AI in language teaching showed that ITS usage had a positive impact on learners in 92% of inspected papers<fnref n="2" /> (Lee and Lee, 2024). Therefore, one should not underestimate the importance of ITS in further developing the field of linguistic education.

ITS has become an umbrella term representing vastly different algorithms: from rule-based approaches to Bayesian Networks to Data Mining methods and Deep Neural Networks (Guo et al., 2021; Mousavinasab et al., 2021). With the grand variety in the field, it may be challenging to give an exact definition to an ITS. Based on details of different systems described in meta-analyses (Pelánek, 2017; Nkambou, Mizoguchi, and Bourdeau, 2010; Stankov, Glavinić, and Rosić, 2000), we consider that an ITS is described by the use of such core components as:

* an **item** (otherwise referred to as **question** or **task** (Pelánek, 2017; Ganter et al., 2017; Sun et al., 2021)): the smallest artifact on which students' knowledge is assessed;

* a **competency** (used interchangeably with **skill** (Düntsch and Gediga, 1995); or **attribute** (DiBello and Stout, 2007)): the learning objective that is tested by an item;

* a **knowledge model** (Pelánek, 2017) or **domain model** (Ganter et al., 2017): a structure representing relations between skills;

<footnote n="1">¹Since the chapter deals with the theoretical background, multiple notions are introduced in its scope. As a guide for the reader, the following guideline is proposed: if a new term is introduced and it will be heavily referred to later in the paper, it is bolded, if deeper discussion of the term lies outside of the scope of the paper, it is used in italics.</footnote>

<footnote n="2">²Seen in 24 papers out of 26.</footnote>
* a **student model**, also referred to as a **learner model** (Pelánek, 2017; Nkambou, Mizoguchi, and Bourdeau, 2010; Desmarais and Baker, 2012): a way of representing learners' transitions between one skill to another.

The systems can differ not only in methodologies used to create them but in the high-level goal they aim to achieve: one can distinguish between *problem solving and solution analysis tutors* and *curriculum sequencing* (Desmarais and Baker, 2012). Problem-solving systems are targeted towards providing feedback to students based on their responses, while curriculum sequencing tutors are utilized for suggesting an appropriate next learning step. Having the project objective in mind, we keep the focus of this section primarily with regards to designing curriculum sequencing applications. In the coming sections of this chapter, we will describe approaches that could be applied to creating the ITS of interest.

## 2.2 Item and Competency Design

Item and competency selection is crucial for the ITS design since it is at the core of such systems. To put it simply, before making any other decisions, the designers of ITS should answer the question: "What do we want our students to learn?"

Frequently this task relies on interviewing experts in the domain and selecting skills based on their feedback, a methodology called **query routine** (Sun et al., 2021). However, other approaches might be applied with the objective of selecting ITS competencies:

* Automatically extracting concepts from texts such as textbooks using Natural Language Processing (NLP) techniques, such as named entity recognition (Guo et al., 2021; Shao et al., 2020)

* Using existing curriculums and their chapter as attributes of ITS (Muangprathub, Boonjing, and Chamnongthai, 2020; Chung and Kim, 2016)

* Employing existing knowledge graphs that could be applied to the domain (Shi et al., 2020; Chen et al., 2021)

Additionally to the plethora of methods to elucidate skills required to build an ITS, the **granularity** of the attributes can also be approached in different ways. Broadly, granularity in this context refers to the level of detail in the selected competencies (Collins, Greer, and Huang, 1996), and it can greatly affect not only usability for students and teachers but also computational complexity (see Figure 2.1) (Pelánek, 2020). The following classification of ITS by the skill granularity was proposed by Culbertson, 2016:

* *Low-level networks:* utilizing high granularity skills in the system design, with sub-categories for a given competence. Low-level networks have to support a complex system that can provide astonishingly precise feedback on the current state of students' mastery, which however, can be overwhelming for instructors to work with (Culbertson, 2016).

* *Mid-level networks:* working with comparatively broader categories. However, it needs to be stated that an exact definition of such a system is not available.

* *High-level networks:* operating with very few categories (as few as 4 subcategories in the field of Information and Communication Technology Literacy
<figure><img src="image_11.png" /><figcaption>FIGURE 2.1: The trade-off between granularity of attributes and coverage of ITS (from Pelánek, 2020).</figcaption></figure>

<table><caption>TABLE 2.1: An example of attribute granularity from Pelánek, 2020.</caption><thead><tr><th>Granularity</th><th>Geography Facts</th><th>Grammar Categories</th></tr></thead><tbody><tr><td>Coarse</td><td>Names and locations of countries</td><td>Capitalization rules</td></tr><tr><td>Medium</td><td>Names and locations of African countries</td><td>Capitalization of nouns</td></tr><tr><td>Fine</td><td>Location of Egypt</td><td>Capitalization of days of the week</td></tr></tbody></table>

Assessment, shown by Almond, Yan, and Hemat, 2007: Create/Communicate, Define/Access, Evaluate, and Manage/Integrate).

It is important to highlight that there is no agreement on what "coarse" or "high level" attributes represent precisely. When comparing the classification from Culbertson, 2016 given above with the attribute examples provided in Table 2.1 (Pelánek, 2020), one can observe that the "coarse" granularity has much more definition, then a high-level network of attributes. For this reason, we propose this nomenclature only to compare the granularity within one work, and not necessarily as a tool for meta-analysis.

Another challenge when designing ITS comes from selecting and designing appropriate items for each of the associated skills. An "ideal question should be informative, trustworthy, and help to identify if students mastered the subject or not" (Collins, Greer, and Huang, 1996). While this definition is quite broad on its own, the mentioned topic of **mastery** is also a vague one that can be defined in a multitude of ways (Beck and Gong, 2013). When interpreting the level of mastery, one might consider **item difficulty** or **item complexity** (Collins, Greer, and Huang, 1996; Pelánek, Effenberger, and Čechák, 2022). It is important to differentiate between the two: item complexity refers to the pre-conceived and contextualized complexity based on expert or system opinion, while item difficulty represents the difficulty of a particular question to students (Pelánek, Effenberger, and Čechák, 2022; Liu and Li, 2012; Beckmann, Birney, and Goode, 2017).

If the aforementioned ambiguities are solved and the appropriate items are selected, they might be linked to the attributes using a **skill map** (Heller et al., 2015; Spoto, Stefanutti, and Vidotto, 2016; Albert, Schrepp, and Held, 1994; Doignon and Falmagne, 2012). Skill map is a triple $(Q, S, \mu)$, where $Q$ is the set of all questions, $S$ is the set of all skills and $\mu$ is a **skill function**:

$$ \forall q \in Q \ \exists \mu(q) = \{C \mid C \subseteq \mathcal{P}(S)\}. $$

If several approaches to solving problems are allowed in the design of the system, the skill function returns all possible combinations of skills that could lead to solving the solution. This type of skill function is called **disjunctive skill function** (Heller et al., 2015). Otherwise, if only one way to provide a solution is expected by the
<table><caption>TABLE 2.2: An example of conjunctive and disjunctive skill mapping of TOEIC questions. Correct answers are bolded. TOEIC questions are taken from Lougheed, 2013.</caption><thead><tr><th></th><th>Mapping type</th><th>TOEIC Question</th><th>Answers</th><th>Skills</th></tr></thead><tbody><tr><td>1</td><td>Conjunctive</td><td>The financial team ___ that the offer was rejected</td><td>A) was disappointing<br>B) were disappointing<br>C) were disappointed<br><b>D) was disappointed</b></td><td>{Agreement of verbs with singular nouns, -ed vs -ing suffixes of adjectives}</td></tr><tr><td>2</td><td>Disjunctive</td><td>___ Ms. Brutus was late, she did not miss the performance.</td><td>A) Because<br>B) During<br>C) In spite of<br><b>D) Although</b></td><td>{Although},<br>{Because, In spite of, During}</td></tr></tbody></table>

ITS, the skill function would return only one subset of competencies required by the problem. Such a type of skill function is called **conjunctive skill function** (find an example of the mapping in Table 2.2). More precisely, here for a correct solution of Question 1 it is expected that students have mastered "-ed" and "-ing" suffixes (i.e. can differentiate between "disappointed" and "disappointing") and understand well agreement of verbs with singular nouns (i.e. know that with "team" one is supposed to use a noun in singular form in English). Since there is just a single expected approach to the solution, such a mapping is conjunctive. On the other hand, for Question 2, one can either master the usage of "although" or solve this task by mastering "because," "during" and "in spite of" and deciding that neither of them work in the given situation.

As one can notice, there exist multiple levels of ambiguities surrounding item and skill design: choosing how to obtain skills, selecting the right granularity, defining the mastery level of skills, and whether or not the system allows for multiple solutions to a question. Despite this fact, the research on the effect of these hyperparameters on ITS performance is still lacking.

## 2.3 Knowledge Model Design

As was shown in the previous section, answering the question regarding the appropriate skill selection for an ITS is a tricky task and at the same time the cornerstone of ITS design. The following decision regarding knowledge model design ((Pelánek, 2017; Nkambou, Mizoguchi, and Bourdeau, 2010)) will heavily rely on the aforementioned selection of skills.

For example, if the set of attributes was derived from expert interviews, one might adapt *rule-based* approaches. This technique involves specialists' definition of an appropriate sequence of topics for students to follow. Essentially, a rule-based ITS would incorporate a series of *IF-THEN* statements: if a student correctly answered Question *a*, they should be proposed Question *b* (Noh et al., 2012). In fact, this type of system design appears to still be prevalent in creating language learning ITS: 2 out of 4 papers regarding language learning ITS designed reviewed by Mousavinasab et al., 2021 were utilizing rule-based approaches. One can rightfully notice that the meta-review does not cover enough language-related ITS, however, we still argue that this sample can be indicative of larger trends in the field. Even though commercial platforms like Babbel<fnref n="3" /> do not fully reveal their algorithms (Loewen, Isbell, and

<footnote n="3">³https://fr.babbel.com/</footnote>
Sporn, 2020), as a user one is expected to complete a series of non-personalized tasks when using these services, which can be associated with rule-based approaches.

While rule-based methodologies are well-suited to represent the expertise of domain specialists and allow for quality control of the questions, the approach is limiting since it can not be applied to larger domains. Additionally, when a personalized approach to teaching is needed, rule-based systems come short (Noh et al., 2012). In this section, we will present some approaches that allow for the creation of a knowledge model using student data.

### 2.3.1 Knowledge Space Theory

One of the techniques to model attributes hierarchy and dependencies of items<fnref n="4" /> is KST (Doignon and Falmagne, 1985; Doignon and Falmagne, 2012; Doignon and Falmagne, 2015; Ganter et al., 2017), which was also called a "fundamental approach in mathematical psychology" (Heck and Noventa, 2020). With this description and the relative commercial success<fnref n="5" /> of the methodology, the theory can be rightfully considered influential in the field of ITS.

To give it a more detailed characterization we need to describe the following core components:

* A **knowledge domain**, which can also be represented as Q or a set of items

* A **knowledge state** $K = \{T \mid T \subseteq Q\}$, which represents a set of questions that can be solved by a student

* A **knowledge structure** represented by $\{\emptyset, K, Q\}$ where K is a set of all possible knowledge states

<figure><img src="image_12.png" /><figcaption>FIGURE 2.2: A representation of a sample knowledge space designed using the knowledge domain from Table 2.3, taken from Ganter et al., 2017.</figcaption></figure>

If $K$ is a closed under union set (meaning $L, M \in K \Rightarrow L \cup M \in K$), such a knowledge structure is called a **knowledge space** (Heck and Noventa, 2020). The

<footnote n="4">⁴The main purpose of KST is to model student performance on given questions (Heller et al., 2015). For these reasons, the core of the methodology lies in modeling items' relations. However, this implementation is limiting since the purpose of education is not questions but attributes mastering (Spoto, Stefanutti, and Vidotto, 2016). This phenomenon served as justification for the introduction of the KST extension: Competence-based KST (CbKST) (Düntsch and Gediga, 1995; Ganter et al., 2017; Heller et al., 2015; Stefanutti et al., 2020). The extension operates using the same principles as the original KST, adding an extra layer of skill maps introduced in Section 2.2.</footnote>

<footnote n="5">⁵KST found application in ALEKS (www.aleks.com), Elektra (https://elektra-project.org/), RATH (https://cognitive-science.at/projects/rath/), Stat-Knowlab (Chiusole et al., 2020), and other commercial systems.</footnote>
<table><caption>TABLE 2.3: An example of knowledge domain for the field of algebra, taken from Ganter et al., 2017</caption><thead><tr><th>Question</th><th>Problem type</th><th>Instance</th></tr></thead><tbody><tr><td>a</td><td>Word problem on proportions</td><td>A car travels on the freeway at an average speed of 52 miles per hour. How many miles does it travel in 5 h and 30 min?</td></tr><tr><td>b</td><td>Plotting a point in the coordinate plane</td><td>Using the pencil, mark the point at coordinates (1, 3)</td></tr><tr><td>c</td><td>Multiplication of monomials</td><td>Perform the following multiplication: 4x<sup>4</sup>y<sup>4</sup> * 2x * 5y<sup>2</sup></td></tr><tr><td>d</td><td>Greatest common factor of two monomials</td><td>Find the greatest common factor of expressions 14<sup>6</sup>y and 4<sup>5</sup>y<sup>8</sup></td></tr><tr><td>e</td><td>Graphing the line through a given point with a given slope</td><td>Graph the line with slope -7 passing through the point (-3, -2)</td></tr><tr><td>f</td><td>Writing the equation of the line through a given point and perpendicular to a given line</td><td>Write an equation for the line that passes through the point (-5, 3) and is perpendicular to the line 8x + 5y = 11</td></tr></tbody></table>

closed under union property of the knowledge space can be better understood with the following example: if there exists a knowledge state {$a, b$} (it is possible to master questions *a* and *b*) and there exists a knowledge state {$c, d$} (it is possible to master questions *c* and *d*), it means that there exists a knowledge state {$a, b, c, d$} (it is possible to master all questions *a*, *b*, *c* and *d*).

Additionally, knowledge spaces are characterized by **surmise** relations between its knowledge states (Brandt, Albert, and Hockemeyer, 2003). We might denote such a relation with *aSc*, which can be read as *a* is surmisable from *c*:

$$aSc \Leftrightarrow \forall K_{c \in K} a \in K_{c \in K}$$

An example of knowledge space modeling can be found in Figure 2.2. Based on the given depiction the surmise relation *aSc* is shown from bottom to the top. From this representation, we can deduce that mastering Question *c* implies previously mastering Question *a*, in other words, Question *a* is a prerequisite for solving Question *c*. More precisely, in the given knowledge space there exist the knowledge states {$a$} and {$a, c$}, however, the state {$c$} where {$a$} is not mastered is deemed not feasible in the system.

Indeed, not all knowledge states are possible. When experts were reviewing all combinatorically possible knowledge states of a knowledge domain *Q*, where $|Q| = 7$, they found that only 34 knowledge states out of 128 ($|K| = 2^7$) were feasible (Dowling, 1993). Therefore, in order to confirm if a knowledge space is feasible, it is often required to get experts' opinion on its validity (Kambouri et al., 1994; Koppen and Doignon, 1990; Koppen, 1993; Schrepp and Held, 1995; Müller, 1989). Nevertheless, this procedure is extremely resource-consuming since even for a small knowledge domain *Q* of 15 questions, the specialists will need to verify $|P(Q)| = 32768$ different knowledge states.

### 2.3.2 KST and Formal Concept Analysis

Since manually ratifying an exponentially large knowledge space is undesirable, one might look for a way to extract observable knowledge states from data using data mining techniques. One such approach is FCA, a methodology whose goal is to uncover relations and hierarchy among attributes and objects (Ganter and Stumme,
2005; Ganter and Wille, 2012). The core of the technique consists of the following parts: a **formal context** and a **concept lattice** representing it.

A formal context K is a triple <G, M, I>, where G is a set of objects, M is a set of attributes<fnref n="6" />, and I ⊆ G × M or *incidence* is a binary relation, representing whether or not a given object has a certain attribute. If an object *g* has an attribute *m*, the expression *gIm* is expected to hold true. A visual representation of a formal context can be found in Table 2.4.

<table><caption>TABLE 2.4: An example formal context for students and the questions from the knowledge domain. We consider that a student possesses an attribute (skill) if they mastered it.</caption><thead><tr><th></th><th>Question<sub>1</sub></th><th>Question<sub>2</sub></th><th>Question<sub>3</sub></th><th>Question<sub>4</sub></th><th>Question<sub>5</sub></th></tr></thead><tbody><tr><th>Student<sub>1</sub></th><td>✓</td><td></td><td>✓</td><td>✓</td><td></td></tr><tr><th>Student<sub>2</sub></th><td>✓</td><td>✓</td><td>✓</td><td></td><td></td></tr><tr><th>Student<sub>3</sub></th><td></td><td>✓</td><td>✓</td><td></td><td></td></tr><tr><th>Student<sub>4</sub></th><td></td><td></td><td></td><td>✓</td><td>✓</td></tr><tr><th>Student<sub>5</sub></th><td>✓</td><td>✓</td><td></td><td></td><td>✓</td></tr></tbody></table>

Given a formal context K, one can establish the following *derivations*:

$$ \bullet \phi(A) = A' = \{m \in M \mid gIm, \forall g \in A\} $$

$$ \bullet \psi(B) = B' = \{g \in G \mid gIm, \forall m \in B\} $$

Additionally when working with FCA, we must define *closure operators* (Ganter and Kuznetsov, 2001) for both extents and intents:

$$ \forall A \subseteq G, A \mapsto A'' = (A')' $$

$$ \forall B \subseteq M, B \mapsto B'' = (B')' $$

If $A' = B$ and $B' = A$, the pair $(A, B)$ is considered to be a *formal concept*, where $A$ can be considered the concept's *extent* (a set of objects, $A \subseteq G$) and $B$ its *intent* (a set of attributes shared by these objects, $B \subseteq M$). One can also refer to set of attributes $B$ as **itemset**, and define **support** of itemset $B$ as:

$$ \mathrm{supp}(B) = \frac{|B'|}{|G|} $$

If $\mathrm{supp}(B) \geq \mathrm{minsup}$ where $\mathrm{minsup}$ is an established threshold, itemset $B$ is called *frequent* (Stumme et al., 2002). The idea behind support and frequent itemset calculation is that not all combinations of attributes would be interesting: if only one student solved a combination of questions {$c, e, d$}, one can consider such combination an outlier and choose to disregard it. As can be observed from the support function definition, $\forall B \subseteq M$ the denominator is a constant $|G|$, therefore, we consider that the following definition of support for the following calculations: $\mathrm{supp}(B) = |B'|$.

Historically, a variety of algorithms has been applied to mining a closed set of itemsets present in the data (Bordat, 1986; Ganter and Kuznetsov, 2001; Nourine and Raynaud, 1999; Kuznetsov, 1993; Jourdan, Rampon, and Jard, 1994; Krajca, Outrata, and Vychodil, 2010; Outrata and Vychodil, 2012). Currently, Linear time Closed itemset Mining, or LCM, (Uno et al., 2003; Uno, Kiyomi, Arimura, et al., 2004) is considered to be the most efficient algorithm in the field (Janostik, Konecny, and

<footnote n="6">⁶Compare with a set of attributes/skills of a knowledge model.</footnote>
Krajča, 2022). As it can be derived from the algorithm's name, the complexity of the algorithm tends to be linear on large datasets which is a significant improvement from the exponential complexity that was previously discussed for KST generation.

Once the concepts are mined, one may form a partially ordered set of those concepts, defined by an operator $\le$ such that $(A_i, B_i) \le (A_j, B_j) \Leftrightarrow A_i \subseteq A_j$ and $(A_i, B_i) \le (A_j, B_j) \Leftrightarrow B_i \supseteq B_j$ (Sarmah, Hazarika, and Sinha, 2015).

One can already notice similarities of the definition above to surmise relations of KST, however, the link is only deepened with the creation of a concept lattice. A concept lattice is partially ordered set characterized by the following property: for each pair of concepts $(A_i, B_i)$ and $(A_j, B_j)$ there exists a unique supremum (also called join) $((A_i \cup A_j)''$, $B_i \cap B_j)$ and a infimum (also referred to as meet) $(A_i \cap A_j, (B_i \cup B_j)''$). Now the similarity between the two theories becomes apparent since they operate on closed sets of attributes (Ganter et al., 2017). As a matter of fact, the link between KST and FCA has been highlighted in multiple research papers (Ganter et al., 2017; Spoto, Stefanutti, and Vidotto, 2010; Liu, 2021; Doignon, 2014).

We will consider the equivalences between KST and FCA that can be found in Table 2.5. As can be noticed, concept lattice and knowledge space can be mapped directly, however, due to the representation notations customary to FCA and KST, the representation of lattice shows {$\emptyset$, $M$} (i.e. state of mastery of all attributes) at the bottom, while KST presents mastery of all elements at the top (an example of a lattice and a corresponding knowledge space can be found in 2.3).

Overall, the mapping of FCA and KST is very advantageous since it allows mining knowledge states directly from data without having to query the experts.

<table><caption>TABLE 2.5: FCA notions and the corresponding KST notions.</caption><thead><tr><th>FCA Notion</th><th>KST Notion</th></tr></thead><tbody><tr><td>Set of attributes <i>M</i></td><td>Set of Questions <i>Q</i></td></tr><tr><td>Set of objects <i>G</i></td><td>–</td></tr><tr><td>Concept lattice <i>K</i></td><td>Knowledge space <i>K</i></td></tr><tr><td>Concept intent <i>B</i></td><td>A knowledge state <i>K</i></td></tr></tbody></table>

## 2.4 Student Modelling

Having modeled the knowledge structure, one would have student modelling as the only remaining part of ITS development. Nevertheless, identifying how students interact with the observed knowledge states is as challenging as the construction of other parts of the system.

One of the ways to model such behavior is to create a **learning path** (Doignon and Falmagne, 2012; Doignon, 2014): a sequence of competencies that students need to master. Interestingly, in the core KST, all learning paths (traversing all states between which there exists a surmise relation) are equally possible (Ganter et al., 2017). In order to address this, an extra probabilistic layer of BLIM (Basic Local Independence Model) (Spoto, Stefanutti, and Vidotto, 2010) is introduced which allows for the prediction of a response *R* of a student, given a knowledge state *K*, where *s* is a probability of a *careless error* and *g* is a probability of a *lucky guess*:

$$P(R | K) = \prod_{i \in K} (1 - s_i)s_i \cdot \prod_{i \notin K} g_i(1 - g_i)$$

<footnote n="7">⁷Set of objects *G* is equivalent to the set of all students observed during training of the model, however, in standard KST the notion is absent.</footnote>
<figure><img src="image_13.png" /><figcaption>FIGURE 2.3: created for the formal context in Table 2.4 and the corresponding knowledge space.</figcaption></figure>

While the notions of lucky guess and careless error can be re-used in different systems, BLIM on its own is created primarily to model students' performance (Heller et al., 2015). If the objective is reversed to elucidate the probability of mastering a knowledge state *K* by the student becomes a complex task, for this reason, reversing the probability to *P(K|R)* requires several assumptions about the distribution of the probability distribution of knowledge states, which might be not desirable especially if not all knowledge states were observed.

Another way to approach the problem is to utilize *Learning Analytics* (LA) (Avella et al., 2016; Chatti et al., 2012). Essentially, LA and data mining are closely related, with the purpose of LA being to extract patterns from students' data (Shemshack, Kinshuk, and Spector, 2021; Avella et al., 2016). The methodology can be applied to cluster students' data and create learner profiles (Govindarajan, Kumar, et al., 2016; Shemshack, Kinshuk, and Spector, 2021). This approach then allows for the creation of a personalizable learning path from student data (Govindarajan, Kumar, et al., 2016), however, this approach might require a lot of data that is frequently unavailable (Silva et al., 2023). For this reason, a combination of approaches like FCA and LA can be used (Ganter et al., 2017).

Like with a lot of topics in the domain of ITS, there is no single industry standard on how learning paths should be constructed, with multiple approaches seeing the light
(Nabizadeh et al., 2020). Additionally, there is no agreement on how the evaluation of constructed learning paths should be evaluated (Nabizadeh et al., 2020; Silva et al., 2023), with most researchers evaluating the results in the *offline* environment: without validating the recommendations in the real world. In this environment, the reproducibility of results of recommended learning paths becomes a problem (Silva et al., 2023).

## 2.5 Other Methodologies

Even though the main focus of this paper lies in utilizing KST for the creation of an ITS, it is necessary to mention other methodologies that can be applied to the task, even if briefly<fnref n="8" />.

One important remark needs to be made regarding the following theories: all of them operate using a type of hidden states that require modeling, unlike KST which operates based on the observed behavior of students (Desmarais and Baker, 2012). Firstly, this allows KST to be more computationally efficient due to the lower number of parameters that need to be calculated. Secondly, it allows us to derive the knowledge structures directly from data.

### Bayesian Knowledge Tracing

The Bayesian Knowledge Tracing (BKT) (Pelánek, 2017; Culbertson, 2016; Corbett, 2000) methodology represents knowledge as states of a Hidden Markov Model, using student responses as observations. It additionally includes the following parameters:

* the probability of initially knowing the skill;

* the probability of learning the skill later;

* the probability of making a mistake when the skill has already been mastered;

* the probability of a lucky guess.

Needless to say, such a model requires the computation of a grand number of probabilistic parameters.

### Item Response Theory

Item Response Theory (IRT) (Samejima, 2012) is a classic theory in the field of ITS where it is assumed that a student has a latent skill irrespectively from the observed answers. Then the relation between latent skill and item response is modeled, using parameters like item difficulty and the probability of a lucky guess. While we can consider both of these notions to be important for an ITS design, incorporating IRT lies outside the scope of the project.

<footnote n="8">⁸Due to the resource limitations of this work, while some theories are introduced in short, others are skipped completely. The following papers are recommended as a starting point for readers who would like to learn more about different approaches that have been used in the field of ITS design: Desmarais and Baker, 2012; Pelánek, 2017; Chrysafiadi and Virvou, 2013</footnote>
# Chapter 3

# Implementation

In this chapter, we will cover the methodology used in our project to create an ITS for English language learning. Section 3.1 will introduce the implemented approach and its steps. Section 3.2 will go through the dataset pruning and annotation process. Section 3.3 will talk about the creation of a knowledge space, while Section 3.4 will describe the algorithm for selecting the best learning path on the created knowledge space. Finally, Section 3.5 will introduce the metrics that were used for evaluating the produced results.

## 3.1 Approach Outline

Before describing the methodology in more detail, it is necessary to provide a general outline of the work done to create our system.

The project was kickstarted by acquiring answers from TOEIC takers and it was utilized as the primary data source. Questions from the dataset were pruned and then mapped with target competencies via a conjunctive skill function, to be used as a basis for automatic extraction of knowledge structure.

As the next step, a mastery function with a threshold (percentage of correct answers to consider a topic learned by a student) was created. Using this function students' answers could be mapped to binary values: whether or not a student knows a given learning objective. These boolean values were used to create a formal context.

Subsequently, the formal context was mined using the LCM algorithm to find all present knowledge states in the data and surmise relations between them, thus creating a knowledge space. Afterwards, we proposed a learning path based on the knowledge space. The recommendations were then evaluated in offline environment using custom metrics: **reproducibility**, **difficulty correlation**, and **complexity correlation**.

All of these steps will be discussed in more detail in coming sections.

## 3.2 Dataset

As was shown in the previous chapter, item and attribute design are crucial for the process of an ITS creation. In our approach, we aimed to stay away from the query routine methodology mentioned above. The main reasoning for this is to avoid introducing extra biases when designing the items and to have attributes that are representative of the data. The dataset in our disposition came from Naimrod's internal resources<fnref n="1" />: it contained answers from 111 university-level students taking 5 different TOEIC exams (500 unique questions, from sample tests of Lougheed, 2013 and

<footnote n="1">¹Due to concerns about the privacy of students and copyright of TOEIC questions the dataset is kept private.</footnote>
Talcott and Tullis, 2007). However, due to the fact that some students did not take all provided tests and/or some tests were taken by very few students, the question set was further reduced to 400 questions and answers from 67 students.

Moreover, considering TOEIC's structure, where the Reading part can be broadly separated into the following parts: Analyzing Vocabulary, Analyzing Grammar, and Analyzing Reading Passages (Lougheed, 2013), we decided to focus only on the Analysing Grammar section. While all the questions are multiple-choice, the latter category heavily relies on reading comprehension of a vast series of texts. Additionally, the Analyzing Vocabulary section might be hard to categorize. Therefore, only one category of questions was kept, making the final dataset contain answers from 67 students for 123 questions, as well as the correct answer for each of the questions.

### 3.2.1 Data Labeling

In order for the dataset to be usable for the final recommendation, each question had to be manually associated with a certain topic of the English language via a skill map.

#### Skill Map

As was shown in Table 2.2, it is frequently possible to solve a TOEIC question using multiple approaches: mastering the language objective that serves as the answer, or mastering the competences that are used as incorrect answers and excluding them. However, in this work we focus on conjunctive mapping only, connecting each question only to the skill that is being tested in the correct answer. It is still possible that multiple competences are required to solve a single item, therefore for the set of questions Q and the set of skills S, the following skill function $\mu(q) : Q \to S$ was utilized within this project:

$$ \forall q \in Q, \exists \mu(q) = \{C | C \subseteq S, C \neq \emptyset.\} $$

$\mu(q)$ is a surjective function (for each skill there exists at least one question testing it), therefore, an inverse **problem function** (Heller et al., 2015) $p(s) : S \to Q$ could be defined:

$$ \forall s, \exists p(s) = \{T | T \subseteq Q, T \neq \emptyset\} $$

The mapped skills were confirmed with an English teacher expert and validated using question types proposed in TOEIC preparation materials (Lougheed, 2013). After the original annotation, we were left with 24 different attributes, however, we experimented with different granularity levels afterward which will be discussed in more detail in Section 4.1.

#### Difficulty Labelling

As mentioned in Section 2.2, the notion of difficulty is important for the assessment of students' skills mastery. For this reason, we wanted to understand the exact difficulty of each attribute for the students in the dataset. In order to achieve this, we introduced the following functions for all students $g \in G^2$ and for all questions $q$ in the pruned set of questions Q:

<footnote n="2">²Here and after we use FCA notation for denoting the set of students (or objects in FCA) as G.</footnote>
* Correctness correct<sub>g</sub>(q) = 
  { 1, if student g picked correct answer
    for question q
  
  0, otherwise

* Success rate sr(q) = $\frac{\sum_{q \in p(s)} correct_g(q)}{|G|}$

Thereafter, a success rate of an attribute sr(s) for s ∈ S was calculated as an average of success rates of all questions that were linked with this skill:

$$sr(s) = \frac{\sum_{q \in p(s)} sr(q)}{|p(s)|}.$$

Finally, each skill s was labeled with difficulty level diff(s) that was observed for the students of our dataset. In our notation, 1 would represent a low difficulty level (or "easy"):

$$diff(s) = rank sr(s).$$

The idea behind the difficulty level labeling is to be able to use it for further evaluation of proposed learning paths: how well do they represent the order of difficulty of topics?

## Complexity Labelling

Along with difficulty, the complexity of skills perceived by domain experts is necessary for understanding students' mastery patterns. The original idea was to send a questionnaire to teachers and propose to sort the skills in the order they would teach them. However, this approach turned out to be unfeasible due to resource limitations, therefore it was decided to turn to existing sources representing knowledge of specialists in the language learning domain. Additionally, this new strategy could help us validate the selected set of attributes (i.e. it is more likely for a skill to be valid if it is present in another dataset).

One of the options for complexity reference could be found in Common European Framework of Reference for Languages (CEFR) 3 which is commonly used to assess student levels, including in tests such as TOEIC. The scale consists of 2 beginner levels (A1 and A2), 2 intermediate levels (B1 and B2), and 2 advanced levels (C1 and C2). These notions are easy to perceive, however, the CEFR rating scale may introduce new biases such as level interpretation depending on the country of origin of the raters (Holzknecht, Huhta, and Lamprianou, 2018), or insufficient precision in the description of competencies (Alderson et al., 2004, as cited by North, 2005). Indeed, the CEFR categories can be hard to interpret and their granularity is extremely coarse as can be seen in Table 3.1.

For this reason, it was decided to use the Global Scale of English (GSE) Teacher Toolkit<fnref n="4" /> instead. The resource contains a vast variety of fine-grained competencies of the English language, each associated with a GSE score ranging from 10 (beginner-level skills) to 90 (advanced-level skills). This evaluation was assigned based on expert evaluation from thousands of teachers from 51 countries <fnref n="5" />.

The interface of the service allows filtering competencies either by keywords or broader topics. This functionality was used to find the competencies related to our

<footnote n="3">³https://rm.coe.int/CoERMPublicCommonSearchServices/DisplayDCTMContent?documentId=090000168045bb52</footnote>

<footnote n="4">⁴https://www.english.com/gse/teacher-toolkit/user/grammar</footnote>

<footnote n="5">⁵https://www.pearson.com/content/dam/one-dot-com/one-dot-com/english/TeacherResources/GSE/GSE-learning-objectives-brochure.pdf</footnote>
<table><caption>TABLE 3.1: CEFR's self-assessment for reading skills in the English language.</caption><thead><tr><th>A1</th><th>A2</th><th>B1</th><th>B2</th></tr></thead><tbody><tr><td>I can understand familiar names, words and very simple sentences, for example on notices and posters or in catalogues.</td><td>I can read very short, simple texts. I can find specific, predictable information in simple everyday material such as advertisements, prospectuses, menus and timetables and I can understand short simple personal letters.</td><td>I can understand texts that consist mainly of high frequency everyday or job-related language. I can understand the description of events, feelings and wishes in personal letters.</td><td>I can read articles and reports concerned with contemporary problems in which the writers adopt particular attitudes or viewpoints. I can understand contemporary literary prose.</td></tr></tbody></table>

attributes. The competencies retrieved from the GSE toolkit were then manually reviewed in order to keep only the competencies exactly covering questions of the sample TOEIC tests. Due to the structure of the source, in some cases, multiple competencies of the GSE Teacher Toolkit could be associated with the same attribute (find an example in Figure 3.1). Therefore, each skill *s* was treated as if it was a set of sub-competencies *c* that were retrieved from the GSE toolkit. The GSE score of each attribute *s* was calculated as an average of the GSE scores of each sub-competency:

$$ gse(s) = \frac{\sum_{c \in s} (gse(c))}{|s|}, $$

where *c* is a subcompetency of skill *s*, *gse*(*c*) is the associated GSE score of the sub-competency. Additionally, the number of retrieved sub-competencies was stored as a way to assess the comparative granularity of attributes.

Finally, similar to the difficulty metric the complexity metric was calculated based on the rank of the GSE scores:

$$ comp(s) = rank(gse(s)^{-1}). $$

To keep the same logic with the difficulty metric, the GSE scores were reversed, making the easiest skills (with the lowest average GSE score) to be associated with the complexity level 1.

As can be seen in Figure 3.1, each of the CEFR levels has an associated GSE threshold. This information was used to map the averaged GSE scores of each attribute back to CEFR levels for easier interpretability.

### 3.2.2 Student Profiles

As mentioned above, the final dataset included answers from 67 students, and all students in the dataset were taking university-level English classes at the time of taking the sample TOEIC tests. One fact important to highlight is that all students had studied English previously. Other information regarding the students' backgrounds was anonymized due to ethical concerns.
<figure><img src="image_14.png" /><figcaption>FIGURE 3.1: The competencies of the GSE Teacher Toolkit associated with attribute *Past Simple*.</figcaption></figure>

However, despite the anonymization we believe, it is crucial to create sub-profiles of students to generate more precise learning path recommendations. For this reason, we separated the students according to their level of English mastery: beginner, intermediate, and upper-intermediate/advanced (further referred to as advanced) students. The profile split was performed using each student's percentage of correct answers: students with 55.5% or less correct answers were assigned into *beginner* group (approx. CEFR level: A1-A2), students with 77.8% or more correct answers were assigned into *advanced* group (approx. CEFR level: B2-C1), and all remaining students were assigned into *intermediate* group (approx. CEFR level: B1)<fnref n="6" />.

<figure><img src="image_15.png" /><figcaption>FIGURE 3.2: Percetange of students corresponding to each student profile.</figcaption></figure>

The decision to assign students who approximately achieved B2 level into the advanced category was driven by the structure of the TOEIC test itself: it does not test C2 level competencies. Additionally, as we can see from the chart in 3.2, despite keeping only B1 level students in the intermediate category, it remained the most populous, so this choice was also an attempt to balance the students' profiles.

<footnote n="6">⁶The thresholds were inspired by the official TOEIC score mapping to CEFR levels: https://etswebsiteprod.cdn.prismic.io/etswebsiteprod/515ff84d-fdd0-4e7f-8d79-e19b4d19e9db_TOEIC+Listening+and+Reading+Mapping+Table+CEFR-092021.pdf.</footnote>
## 3.3 Knowledge Space Creation

With the cleaned and annotated data, one could move towards mining knowledge states and relations between them. Our approach to designing an ITS includes utilizing the aforementioned link between KST and FCA, which allows for automatically mining knowledge states and relations between them.

### 3.3.1 Converting Answers to Formal Context

As the first stepping stone towards FCA, one must create a binary formal context. With this purpose, we defined a mastery function:

$$m(s,g) = \begin{cases} 1, & \text{if } \frac{\sum_{\forall q \in p(s)} correct_g(q)}{|p(s)|} \ge \theta, \\ 0, & \text{otherwise} \end{cases}$$

where $\theta$ is a **mastery threshold**. The mastery threshold can range from 0 to 100%, and it represents the expected percentage of correct answers that should correspond to mastery of the topic.<fnref n="7" /> A slice of the formal context that was formed after applying the mastery function to $S \times G$ can be found in Table 3.2. Additionally to it, we created subcontexts for different student profiles, instead of G using $G_{beginners}$, $G_{intermediate}$, and $G_{advanced}$.

<table><caption>TABLE 3.2: Binary context $S \times G$ representing mastery of skills by students.</caption><thead><tr><th></th><th>Student g<sub>1</sub></th><th>Student g<sub>2</sub></th><th>...</th><th>Student g<sub>n</sub></th></tr></thead><tbody><tr><td>Present Simple</td><td>1</td><td>1</td><td>...</td><td>0</td></tr><tr><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td></tr><tr><td>Suffixes of adjectives</td><td>0</td><td>1</td><td>...</td><td>0</td></tr></tbody></table>

Each column can be also read as $\vec{v}_g$, where

$$\vec{v}_g = \begin{bmatrix} m(s_1, g) \\ m(s_2, g) \\ \vdots \\ m(s_{|S|}, g) \end{bmatrix},$$

for all $g \in G$, $s_i \in S$, that is also a representation of a student's detected knowledge state $K_g = \{s_i | m(s_i, g) = 1, i \in 1...|S|\}$. However, one can guess that the binary context alone is not a good representation of a knowledge space: in this project, only 67 states<fnref n="8" /> were observed out of $2^{24} = 16777216$ possible. For this reason, the context is then mined to identify frequently seen subsets of skills among the already observed knowledge states.

### 3.3.2 Mining Frequent Itemsets

Frequent itemset mining (or frequent pattern mining (Zaki and Hsiao, 2002)) should be understood as mining frequently observed sequences. Due to the binarized representation of $\vec{v}_g^T$, we can look for commonly observed sequences of 0s and 1s. If such sequences are found, they are treated as knowledge states the students had to

<footnote n="7">⁷If there is no tolerance for lucky guesses, a 100% mastery threshold needs to be adapted.</footnote>

<footnote n="8">⁸It is also worth highlighting that observed knowledge states of some students were identical.</footnote>
pass in order to arrive at their currently observed knowledge state. Such subsets $s'$ are added to the knowledge space $\mathcal{K}$, as well as $s' \cup \mathcal{K}$, for all $\mathcal{K} \in \mathcal{K}$, thus ensuring that the set of knowledge states remains closed under union.

LCM was the algorithm utilized for the itemset mining, more specifically its implementation of caspaileur<fnref n="9" />. The methodology was used as black-box<fnref n="10" />, with the only modifying parameter being itemset frequency: in our case, the itemsets with the support $\ge 2$ were added to the original knowledge space, thus creating the extended knowledge space $\mathcal{K}'$. Depending on hyperparameters discussed in more detail in Chapter 4, the size of obtained knowledge spaces varied from dozens to tens of thousands. Therefore, the extended knowledge spaces were still significantly smaller than $2^{|\mathcal{S}|}$ which made them more computationally sustainable in addition to being representative of the data.

### 3.3.3 Mining Surmise Relations

In order to be able to use knowledge space for learning path recommendations, one must identify surmise relations present in the data. Since LCM implementation returns all observed frequent itemsets as a sorted array, it is straightforward to traverse this data structure. With the purpose mine all surmise relations of the knowledge space $\mathcal{K}'$, we use a naive approach described in the Algorithm 1¹¹.

<caption>**Algorithm 1 Mining surmise relations**</caption>

Require: Ordered array of knowledge states $\mathcal{K}'$
$n \leftarrow |\mathcal{K}'|$
$\mathcal{M} \leftarrow \text{Matrix (size: } n \times n)$
$\quad \text{for } i \in [1 \dots n] \text{ do}$
$\quad \quad \text{for } j \in [1 \dots n] \text{ do}$
$\quad \quad \quad \text{if } \mathcal{K}'[i] \subset \mathcal{K}'[j] \text{ then}$
$\quad \quad \quad \quad \mathcal{M}[i][j] = 1$
$\quad \quad \text{end if}$
$\quad \quad \text{end for}$
$\quad \quad \text{end for}$
$\quad \quad \text{return } \mathcal{M}$

$\quad \text{Store all existing surmise relations in a matrix}$

$\quad \text{If there is no subset relation between knowledge states, there can not exist a surmise relation between them}$

$\quad \text{Since } \mathcal{K}'[i] \text{ is a subset of } \mathcal{K}'[j],$ one must master it $\mathcal{K}'[i]$ to master $\mathcal{K}'[j]$, therefore, there is a surmise relation between the states.

## 3.4 Building a Learning Path

As a result of running Algorithm 1, we obtain matrix $\mathcal{M}$ that can be interpreted as edges of a directed graph. Knowledge space as a whole can be viewed as such:

<footnote n="9">⁹https://github.com/EgorDudyrev/caspailleur/</footnote>

<footnote n="10">¹⁰LCM implementation details can be found in Uno et al., 2003; Uno, Kiyomi, Arimura, et al., 2004; Janostik, Konecny, and Krajča, 2022.</footnote>

<footnote n="11">¹¹As it can be observed, the algorithm is not optimal with the complexity of $O(n^2)$, due to relatively small size of the knowledge space, the performance of such algorithm was satisfactory. However, for larger spaces a more optimal algorithm should be considered.</footnote>
knowledge states are none other than directed graph nodes, and surmise relations are directed edges. To be more precise, if there exists a surmise relation $K_iSK_j$, one can represent it equally as two nodes $K_i$ and $K_j$ and a directed edge $(K_i, K_j)$.

This reinterpretation gives us a lot of freedom, since using this approach one can assign additional weights to whatever surmise relations are deemed more important.

### 3.4.1 Learning Edge Weights

In our approach, we wanted the weights of edges to represent the frequency of surmise relations. One could assume that an appropriate way to represent this information is to assign a weight equal to the number of students who mastered both $K_i$ and $K_j$. First, the supports of all knowledge states need to be calculated (see Algorithm 2 for more details).

<caption>**Algorithm 2** Calculate edge weights</caption>

**Require:** A square matrix $\mathcal{M}$ of directed edges
**Require:** A formal context $\mathcal{K}$
**Require:** Set of students $G$
$n \leftarrow |\mathcal{M}[0]|$
**for** $i \in [1 \dots n]$ **do**
&emsp;&emsp;**for** $j \in [1 \dots n]$ **do**
&emsp;&emsp;&emsp;&emsp;**if** $\mathcal{K}'[i] \subset \mathcal{K}'[j]$ **then**
&emsp;&emsp;&emsp;&emsp;&emsp;$$ \mathcal{M}[i][j] = |\{g \mid g \in G \text{ and } m(s,g) = 1, \forall s \in \mathcal{K}'[j]\}| $$
&emsp;&emsp;&emsp;&emsp;**▷** If a directed edge exists, assign a new weight.
&emsp;&emsp;&emsp;&emsp;**▷** Since $\mathcal{K}'[i]$ is a prerequisite of $\mathcal{K}'[j]$, the number of students who mastered both is equal to number of students who mastered of $\mathcal{K}'[j]$.
**end if**
**end for**
**end for**
**return** $\mathcal{M}$

However, one of the main concerns when constructing an ITS is its reproducibility on unseen data. Therefore, the approach was adjusted to learn weights so that the most frequently seen surmise relations are mapped to higher weights of edges. Since it is not attainable for us to obtain large datasets allowing real training of weights on unseen data we opted to use *k-fold* (Dietterich, 1998) splits with $n^{12}$ repetitions<fnref n="13" /> To be exact, the following approach was followed:

1. Split $G^{14}$ into $k = 2$ parts randomly.

2. Assign one split to be train<fnref n="15" /> context and the other test context.

<footnote n="12">¹²In all our experiments $n = 30$.</footnote>

<footnote n="13">¹³In its scikit-learn implementation: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RepeatedKFold.html#sklearn.model_selection.RepeatedKFold.</footnote>

<footnote n="14">¹⁴The same approach is followed by student profile specific contexts, using $G_{beginner}$, $G_{intermediate}$, $G_{advanced}$.</footnote>

<footnote n="15">¹⁵Here and after notions *train* and *test* are used frivolously. There is no training process per se, *train* name is used to signify that train weights are stored between the runs and are updated after the run.</footnote>
3. Find knowledge states for test (see 3.3.2).

4. Find surmise relations of knowledge states for test (see Algorithm 1).

5. Calculate edge weights for test (see Algorithm 2).

6. Add unseen nodes $K \in K_{test} \setminus K_{train}$ to $K_{train}$ and add associated edges with weight equal to 0.

7. Update $weight(K_i, K_j)_{train} = (weight(K_i, K_j)_{train} + weight(K_i, K_j)_{test})/2$ if $\exists (K_i, K_j)_{test}$.

8. Propose a learning path based on the weights (see Section 3.4.2).

9. Evaluate the proposed path (see Section 3.5).

10. Repeat Steps 1-9 for $n-1$ times.

A small example of how Steps 6-7 were applied can be found in Figure 3.3. As shown in Figure 3.3a, during the first splitting the train graph is initiated as an empty set (without any nodes or edges). After implementing Step 7 in the first iteration, the unseen nodes are added to the train graph, and the weights of edges are averaged. During iteration 2 which can be seen in Figure 3.3b, the train graph is initiated reusing the updated graph from the previous step. Similarly to the previous step unseen nodes and edges are added and initiated with 0s, while the edges that are seen in the test graph are averaged.

<figure><img src="image_16.png" /><figcaption>FIGURE 3.3: An example of weight update during the 2-fold splitting with $n$ repetitions.</figcaption></figure>

### 3.4.2 Learning Path from Learned Weights

Having produced a graph similar to what can be observed in the example from Figure 3.3b, one can utilize it to propose the best learning path. Since the main reasoning for such an approach with training weights of edges was to find the most frequent transitions, a greedy algorithm is proposed to construct the best learning path (see Algorithm 3). The idea behind this algorithm is to start at zero knowledge, pick the node connected to it with the edge of the highest weight, and continue repeating the same process until arriving at the mastery of all skills.
**Algorithm 3 Greedy algorithm for the best learning path**

**Require:** A square matrix $\mathcal{M}$ of directed edges with trained weights

**Require:** An array of knowledge states $\mathcal{K}$

**Require:** Set of skills $S$

$n \leftarrow |\mathcal{M}[0]|$

$path \leftarrow []$

$step \leftarrow \emptyset$

$i \leftarrow$ index of step in array $\mathcal{K}$

**while** step $\neq$ S **do**:

$i \leftarrow \text{argmax}_j(\mathcal{M}[i][j])$ for $j \in [1...n]$

$step \leftarrow \mathcal{K}[i]$

$path \leftarrow path \cup step$

**end while**

**return** path

### 3.4.3 Building a Baseline Learning Path

As can be observed from the previous sections, the algorithms behind training the weights of edges and mining surmise relations are relatively computation-heavy. Therefore, one can rightfully ask whether this type of approach is justifiable if a learning path can be created with a much more lightweight method. For this reason, we introduced a baseline learning path created by maximizing conditional probabilities of each transition on the data.

$$step_i = \text{argmax}_s \left( \frac{|m(s,g) \wedge m(step_{i-1,g})|}{|m(s,g)|} \right)$$

Only formal context and the set of skills are utilized to produce such a path, making it much more efficient than the proposed algorithm utilizing trained weights. The baseline algorithm description can be found in Algorithm 4.

**Algorithm 4 Conditional probability-based learning**

**Require:** A formal context $\mathcal{K}$

**Require:** Set of skills $S$

**Require:** Set of students $G$

$n \leftarrow |S|$

$path \leftarrow []$

$step \leftarrow \emptyset$

$i \leftarrow$ index of step in array $S$

**while** step $\neq$ S **do**:

$i \leftarrow \text{argmax}_i \left( \frac{\sum_{v_g \in G} m(s_i, g)}{\sum_{v_g \in G} m(step, g)} \right)$ for $i \in [1...n]$

$step \leftarrow \mathcal{K}[i]$

$path \leftarrow path \cup step$

**end while**

**return** path

Similarly to the approach with the weighted graph, we utilized k-fold splits with repetitions, in the following manner:

1. Split G into $k = 2$ parts randomly.
2. Assign one split to be train context and the other test context<fnref n="16" />.

3. Using the train context and Algorithm 4, create a proposed path.

4. Evaluate the proposed path (see Section 3.5).

5. Repeat Steps 1-4 for $n-1$ times.

One might have proposed an even simple baseline path: attributes ranked by their difficulty and/or complexity, however, due to the low correlation of difficulty and complexity in the dataset (Spearman's $\rho = 0.3$), we decided not to follow this approach since we deemed this correlation too low to serve as a good baseline.

### 3.4.4 Path Post-Processing

In order to be able to make correct evaluations, the produced paths from both the trained and baseline approaches had to be post-processed. Firstly, due to the nature of knowledge space, the proposed paths were traversing all unique knowledge states and had format `[{a}, {a,b}, {a,b,c} ...]`.

However, we were only interested in new skills that students have to learn, therefore, all paths were converted into sequences of only new attributes, not mastered at any previous step of the path:

$$[{a}, {a,b}, {a,b,c} ...] \rightarrow [{a}, {b}, {c} ...]$$

Additionally, we calculated

$$path_order(s, path) = \{i \mid s \in path[i], i \in [0...|path|]\},$$

to be able to make easier comparisons between different sequences of attributes.

## 3.5 Evaluation

Above we described an approach to how an ITS can be created to generate a sequence of topics suggested for students to learn, however, it is complicated to assess whether or not these recommendations are valid. Due to the project limitations, the proposed learning paths could not be evaluated via a user study. For this reason, it was decided to create a series of offline metrics that would answer the following questions:

* How well does the learning path reflect the difficulty of skills?

* How well does the learning path reflect the complexity of skills?

* How well can the learning path apply to unseen data?

For all the metrics we chose to use Spearman's $\rho$ (Spearman, 1961) since it was specifically tailored to compare rankings. Additionally, a correlation metric is easily interpretable with the desirable outcome to be expected around 0.8-1. Moreover, due to the same computation methodology, one can aggregate (for example, average) all introduced metrics to get an understanding of the overall performance of the system.

<footnote n="16">¹⁶The splitting is used for comparing the behavior of the baseline approach when faced with unseen data.</footnote>
### 3.5.1 Difficulty Correlation

Difficulty correlation is a proposed way to assess how well the recommended learning paths correspond to the actual difficulty of attributes for students in the dataset.

Two vectors were calculated: $\vec{p}_{\text{path}} = \begin{bmatrix} \text{path\_order}(s_1, \text{path}) \\ \text{path\_order}(s_2, \text{path}) \\ \vdots \\ \text{path\_order}(s_{|S|}, \text{path}) \end{bmatrix}$ and $\vec{d} = \begin{bmatrix} \text{diff}(s_1) \\ \text{diff}(s_2) \\ \vdots \\ \text{diff}(s_{|S|}) \end{bmatrix}$.

Computations of $\vec{d}$ are independent of the learning path and knowledge space and are only required once. Therefore, the final metric representing difficulty correlation was computed in the following manner:

$$ \text{diff\_corr}(\text{path}) = \rho(\vec{p}_{\text{path}}, \vec{d}) $$

### 3.5.2 Complexity Correlation

The approach towards correlation with complexity perceived by experts was calculated in the same manner as difficulty correlation. The only difference would be the computation of the vector $\vec{c}$:

$$ \vec{c} = \begin{bmatrix} \text{comp}(s_1) \\ \text{comp}(s_2) \\ \vdots \\ \text{comp}(s_{|S|}) \end{bmatrix}, $$

making the final complexity correlation metric:

$$ \text{comp\_corr}(\text{path}) = \rho(\vec{p}_{\text{path}}, \vec{c}) $$

### 3.5.3 Reproducibility

As voiced above, the reproducibility of the ITS results is one of the main issues we tried to address with the proposed approach of training weights of graph edges. For this reason, albeit somewhat similar to the metrics above, the reproducibility metric heavily relies on train/test split. The ability of a learning path to generalize is computed using this formula:

$$ \text{reproducibility}(\text{path}_{\text{train}}, \text{path}_{\text{test}}) = \rho(\vec{p}_{\text{path}_{\text{train}}}, \vec{p}_{\text{path}_{\text{test}}}) $$
# Chapter 4

# Experiments

Having discussed the methodology given in Chapter 3, in this chapter, we will cover the experiments regarding the setup: what kind of hyperparameters are worth considering when designing an ITS?

In Section 4.1 we will talk about the attribute granularity experiment, Section 4.2 will present the effect of applying different mastery thresholds, and Section 4.3 will demonstrate the impact of questions difficulty on the performance of an ITS. In the course of these sections, we will pick the best-performing parameters and utilize them to compare the results of the trained path with the baseline approach in Section 4.4.

## 4.1 Granularity of Attributes

The first question that we decided to ask ourselves "How the attribute granularity affects ITS performance?" In order to answer this question, one must treat the skill map function $\mu(q)$ as a hyperparameter. Indeed, changing the $\mu(q)$ definition affects its inverse $p(s)$ which in turn is used for the calculation of the mastery function that lies at the core of further computations.

In our set up we introduced $\mu_{coarse}(q) : Q \mapsto S_{coarse}$ and $\mu_{finer}(q) : Q \mapsto S_{finer}$. Note that we refer to the granularity level as *coarse* and *finer*, in order to be able to compare the two, however, it should not be interpreted that *coarse* category is the coarsest possible in the domain of English language. Similarly, we call the other set of attributes *finer* as a way to distinguish the two. Still, at this stage, it is impossible to define where exactly the border between coarse and fine-grain skill definition lies.

<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /><figcaption>FIGURE 4.1: The percentage of questions for each CEFR level split by granularity of attributes.</figcaption></figure>
In total, we found 24 coarse and 35 finer attributes<fnref n="1" /> to connect with the questions of set Q. As it appears in 4.1, finer attributes are significantly more heterogeneous in their structure compared to coarse ones. More precisely, the percentage of questions of CEFR's B1.1 level drops from 48.3% to just 26.3% when comparing these granularities. Along with this, all other CEFR categories become more present in the data (see 4.1b). This point can be further developed when analyzing Table 4.1: even though the standard deviation of the number of sub-competencies is still high for finer attributes, it very notably dropped compared to coarse attributes. This could be a signal that finer attributes have a much more similar level of granularity among themselves.

<table><caption>TABLE 4.1: Standard deviations of the number of sub-competences, GSE score, and the number of questions per attribute, using coarse and finer attributes.</caption><thead><tr><th>Granularity</th><th># Attributes</th><th>Std. of # Sub-competencies</th><th>Std. of GSE score</th><th>Std. of Questions</th></tr></thead><tbody><tr><td>Coarse</td><td>24</td><td>20.86</td><td>7.12</td><td>3.83</td></tr><tr><td>Finer</td><td>35</td><td>12.31</td><td>9.54</td><td>1.62</td></tr></tbody></table>

On the contrary, as shown in Table 4.1, the standard deviation for gse(s) has increased. Along with Figure 4.1a, this points again towards the highly homogeneous structure of coarser attributes. If one further investigates this, it is possible to find that coarser attributes lead to smoother distribution due to averaging a sizable amount of different questions. As can be seen in Table 4.2, a coarse attribute "Verb Agreement" includes in itself a multitude of different finer categories: agreement of verb with the subject, agreement of infinitive with modal verbs, usage of -ing or infinitive forms after a verb, subjunctive, causative agreement. Due to the finer grain of skills, we can uncover that despite having lower complexity "Modal Verbs" have significantly lower success rates than "Subjunctive". This could illustrate the fact that the students are French speakers familiar with the subjunctive concept, while the complexity of the topic reflected in the GSE score and the corresponding CEFR level do not take into account the linguistical background of the learner.

<table><caption>TABLE 4.2: Average success rates per "Verb Agreement" attributes. The coarse attribute is shown before the bold line and the corresponding finer attributes can be found after the bold line.</caption><thead><tr><th></th><th>[B1.1]<br>Verb<br>agreement</th><th>[A1]<br>Subject &<br>verb</th><th>[B1.1]<br>Modal verbs</th><th>[B1.2]<br>-Ing or<br>infinitive</th><th>[B1.2]<br>Subjunctive</th><th>[B1.2]<br>Causative</th></tr></thead><tbody><tr><td>All students</td><td>56.2%</td><td>65.4%</td><td>50.7%</td><td>60.6%</td><td>58.7%</td><td>35.8%</td></tr><tr><td>Beginners</td><td>36.1%</td><td>44.3%</td><td>26.1%</td><td>44.3%</td><td>39.1%</td><td>13.0%</td></tr><tr><td>Intermediate</td><td>58.9%</td><td>70.7%</td><td>48.2%</td><td>63.6%</td><td>65.5%</td><td>35.7%</td></tr><tr><td>Advanced</td><td>80.1%</td><td>86.3%</td><td>90.6%</td><td>78.8%</td><td>75.0%</td><td>68.8%</td></tr></tbody></table>

In order to further compare different granularity levels and their effect on system design, we trained the weights of a graph with the following parameters:

* $n = 30$

* $\theta = 100\%$

* $p(s) = \begin{cases} \mu_{coarse}(q)^{-1}, & \text{for } S_{coarse} \\ \mu_{finer}(q)^{-1}, & \text{for } S_{finer} \end{cases}$

<footnote n="1">¹Find the list of finer-grained attributes in Appendix A</footnote>
If we are to compare the metrics' results for learning paths created using coarse and finer attributes for all students, we can notice no clear trends (see Figure 4.2). This can further be confirmed using a t-test, with the null hypothesis that coarse and finer attributes produce the same metrics<fnref n="2" />. By setting $p-value < 0.05$ we can not reject any of the null hypotheses (see Table 4.3), therefore, we might conclude the metric results for different granularity of attributes are very comparable. Therefore, picking finer or coarser attribute grains might not have as strong an impact as expected.

<table><caption>TABLE 4.3: P-values for t-test performed comparing metric results for all students using coarse- and finer-grained attributes. The p-value threshold is set to be $p-value \le 0.05$.</caption><thead><tr><td>Metric</td><td>P-value</td></tr></thead><tbody><tr><td>Reproducibility</td><td>3.5 * 10<sup>-1</sup></td></tr><tr><td>Difficulty correlation</td><td>6.2 * 10<sup>-2</sup></td></tr><tr><td>Complexity correlation</td><td>4.6 * 10<sup>-1</sup></td></tr></tbody></table>

<figure><img src="image_19.png" /><figcaption>(A) Reproducibility</figcaption></figure>

<figure><img src="image_20.png" /><figcaption>(B) Difficulty correlation</figcaption></figure>

<figure><img src="image_21.png" /><figcaption>(C) Complexity correlation</figcaption></figure>

<figure><img src="image_22.png" /><figcaption>(D) Aggregated metrics</figcaption></figure>

<caption>FIGURE 4.2: Metric results for coarse and finer attributes, aggregated (averaged) for different student groups. Figure 4.2d represents the aggregated (averaged) results of all 3 metrics, with tinted areas representing the standard deviation of results.</caption>

However, both visually from Figure 4.2b and from the t-test results, it appears that difficulty correlation performs slightly better when using coarse granularity in comparison to finer ones. One explanation for such results would be the aforementioned more homogenous inner structure of coarse attributes.

<footnote n="2">²The same null hypothesis will be used in the coming sections.</footnote>
It is also worth noting that for both approaches reproducibility and difficulty correlation performed stably, with 2 metrics varying around 0.6 (which should be considered relatively good for correlation results), while the complexity correlation returns much lower results (0.3 at best), and also has a significantly higher standard deviation. This could be explained by the fact that the complexity of attributes is not representative of how students learn, especially since they learned English before (see Table 4.2).

Even though both granularities have shown very similar performance, the finer granularity was deemed more desirable for our ITS, therefore, for the following experiments only the finer skill map was utilized.

## 4.2 Comparing Mastery Thresholds

Another question frequently posed during an ITS design is: "What mastery threshold is appropriate? What percentage of lucky guesses is allowed?" As was shown in Calò, 2021, threshold design does not guarantee a better correlation with the perceived complexity of attributes. In our experiments, we want to tackle the question in a slightly different way: we argue that the threshold design is also dependent on the number of questions used per attribute, that setting $\theta = 100\%$ on 3 questions is different from setting $\theta = 100\%$ on 4 questions.

For this experiment design the following protocol was applied:

* $n = 30$

* $(\theta, p(s)) = \begin{cases} (100\%, p(s)[: 3]), & \text{Apply 100\% mastery threshold on 3 questions} \\ (100\%, p(s)[: 4]), & \text{Apply 100\% mastery threshold on 4 questions} \\ (66\%, p(s)[: 3]), & \text{Apply 66\% mastery threshold on 3 questions} \\ (75\%, p(s)[: 4]), & \text{Apply 100\% mastery threshold on 4 questions} \end{cases}$

The metrics regarding generated learning paths when using different thresholds can be found in Figure 4.3. One can notice in Figure 4.3d, that choosing 3 questions with 100% mastery appears to be the worst performant approach, however, it is hard to interpret other trends from the graphs. For this reason, once again we turn to a t-test to compare the metric results of approaches using different thresholds (see Table 4.4). From the results, we can deduce that using 3 questions with 100% mastery is statistically significantly less performant than choosing 3 questions with 66% mastery or 4 questions with 75% mastery. It could be that mastery thresholds that allow for careless error are associated with higher reproducibility rates, however, this needs to be confirmed in further research.

As far as our experiments go, none of the approaches to defining thresholds seem to affect difficulty correlation results (see Table 4.4c). However, contrary to Calò, 2021, we observed some statistically significant differences for complexity correlation: for example, 3 questions with 66% mastery performed better than 3 questions with 100% mastery and 4 questions with 75% mastery. This can be explained that for both of these underperforming approaches, we expect students to solve at least 3 questions correctly, and when compared with allowing for a higher careless error rate these setups seem to yield worse results. However, when seeing the same trend for choosing 4 questions with 100% mastery, it is required to continue the investigation.

For this reason, we decided to compare the aggregated metric performance by student profile (see Figure 4.4). As one can notice for the beginner (Figure 4.4b)
<figure><img src="image_23.png" /><figcaption>(A) Reproducibility</figcaption></figure>

<figure><img src="image_24.png" /><figcaption>(C) Complexity correlation</figcaption></figure>

<figure><img src="image_25.png" /><figcaption>(B) Difficulty correlation</figcaption></figure>

<figure><img src="image_26.png" /><figcaption>(D) Aggregated metrics</figcaption></figure>

<table><caption>FIGURE 4.3: Metric results for different threshold approaches, aggregated (averaged) for different student groups. Figure 4.3d represents the aggregated (averaged) results of all 3 metrics, with tinted areas representing the standard deviation of results.</caption><thead><tr><th></th><th>4 questions, t=100</th><th>4 questions, t=75</th><th>3 questions, t=66</th><th>3 questions, t=100</th></tr></thead><tbody><tr><th>4 questions, t=100</th><td>1.0 (0)</td><td>6.4 * 10<sup>-2</sup> (-1.888)</td><td>3.5 * 10<sup>-1</sup> (-0.944)</td><td>3.6 * 10<sup>-1</sup> (0.927)</td></tr><tr><th>4 questions, t=75</th><td>6.4 * 10<sup>-2</sup> (1.888)</td><td>1.0 (0)</td><td>3 * 10<sup>-1</sup> (1.057)</td><td>5 * 10<sup>-3</sup> (2.907)</td></tr><tr><th>3 questions, t=66</th><td>3.5 * 10<sup>-1</sup> (0.944)</td><td>3 * 10<sup>-1</sup> (-1.057)</td><td>1.0 (0)</td><td>5 * 10<sup>-2</sup> (1.967)</td></tr><tr><th>3 questions, t=100</th><td>3.6 * 10<sup>-1</sup> (-0.927)</td><td>5 * 10<sup>-3</sup> (-2.907)</td><td>5 * 10<sup>-2</sup> (-1.967)</td><td>1.0 (0)</td></tr></tbody></table>

<table><caption>(A) Reproducibility</caption><thead><tr><th></th><th>4 questions, t=100</th><th>4 questions, t=75</th><th>3 questions, t=66</th><th>3 questions, t=100</th></tr></thead><tbody><tr><th>4 questions, t=100</th><td>1.0 (0)</td><td>6.3 * 10<sup>-1</sup> (-0.484)</td><td>4.7 * 10<sup>-1</sup> (0.727)</td><td>2.8 * 10<sup>-1</sup> (1.095)</td></tr><tr><th>4 questions, t=75</th><td>6.3 * 10<sup>-1</sup> (0.484)</td><td>1.0 (0)</td><td>2.4 * 10<sup>-1</sup> (1.196)</td><td>1.3 * 10<sup>-1</sup> (1.535)</td></tr><tr><th>3 questions, t=66</th><td>4.7 * 10<sup>-1</sup> (-0.727)</td><td>2.4 * 10<sup>-1</sup> (-1.196)</td><td>1.0 (0)</td><td>6.9 * 10<sup>-1</sup> (0.406)</td></tr><tr><th>3 questions, t=100</th><td>2.8 * 10<sup>-1</sup> (-1.095)</td><td>1.8 * 10<sup>-1</sup> (-1.535)</td><td>6.9 * 10<sup>-1</sup> (-0.406)</td><td>1.0 (0)</td></tr></tbody></table>

<table><caption>(B) Difficulty correlation</caption><thead><tr><th></th><th>4 questions, t=100</th><th>4 questions, t=75</th><th>3 questions, t=66</th><th>3 questions, t=100</th></tr></thead><tbody><tr><th>4 questions, t=100</th><td>1.0 (0)</td><td>6.9 * 10<sup>-6</sup> (4.941)</td><td>7.8 * 10<sup>-1</sup> (-0.282)</td><td>2.4 * 10<sup>-7</sup> (5.844)</td></tr><tr><th>4 questions, t=75</th><td>6.9 * 10<sup>-6</sup> (-4.941)</td><td>1.0 (0)</td><td>4.5 * 10<sup>-8</sup> (-6.289)</td><td>6.2 * 10<sup>-1</sup> (0.498)</td></tr><tr><th>3 questions, t=66</th><td>7.8 * 10<sup>-1</sup> (0.282)</td><td>4.5 * 10<sup>-8</sup> (6.289)</td><td>1.0 (0)</td><td>1 * 10<sup>-10</sup> (7.832)</td></tr><tr><th>3 questions, t=100</th><td>2.4 * 10<sup>-7</sup> (-5.844)</td><td>6.2 * 10<sup>-1</sup> (-0.498)</td><td>1 * 10<sup>-10</sup> (-7.832)</td><td>1.0 (0)</td></tr></tbody></table>

<caption>(C) Complexity correlation</caption>

<caption>TABLE 4.4: T-test results (p-value and t-statistic) from comparing metric results for different threshold approaches. The p-value threshold is set to be *p* - *value* ≤ 0.05.</caption>
and for the intermediate student (Figure 4.4c) groups, setting the threshold for 4 questions with 100% mastery appears to be one of the worst performing approaches, however, for advanced students (Figure 4.4d) this type of approach becomes the best performant. Therefore, one can conclude that choosing an appropriate threshold depends on the target student profile, and selecting a threshold with zero tolerance for accidental mistakes might be detrimental to beginner and intermediate students, while it appears to model well the learning process of advanced students.

<figure><img src="image_27.png" /><figcaption>(A) Aggregated metrics for all students</figcaption></figure>

<figure><img src="image_28.png" /><figcaption>(C) Aggregated metrics for intermediate students</figcaption></figure>

<figure><img src="image_29.png" /><figcaption>(B) Aggregated metrics for beginners</figcaption></figure>

<figure><img src="image_30.png" /><figcaption>(D) Aggregated metrics for advanced students</figcaption></figure>

FIGURE 4.4: Aggregated (averaged) metric results for different thresholds, grouped by student profile.

Since advanced students are underrepresented in the dataset, we decided not to go with the threshold of 4 questions with 100% and go with 3 questions and 66% mastery instead, since it appears to be well performant on all metrics and yields good results for beginners and intermediate students.

## 4.3 Comparing Difficulty of Questions in Attributes

Now that we identified that for our experiment it is better to select 3 questions and set the threshold to 66% mastery, one might ask: "What is the best approach to choose these 3 questions? Should we select the easiest or the hardest questions?"

In order to answer these questions we design the following experiment:
$$ \bullet p(s) = \begin{cases} p(s), & \text{As the baseline use all questions associated with the skill.} \\ \{\operatorname{argmax}_q sr(q)\}[:3] | s \in \mu(q), & \text{Pick 3 questions with the highest success rate (easiest) questions.} \\ \{\operatorname{argmin}_q sr(q)\}[:3] | s \in \mu(q), & \text{Pick 3 questions with the lowest success rate (hardest) questions.} \\ \{\operatorname{argmin}_q (sr(q) - \operatorname{diff}(s))\}[:3] | s \in \mu(q), & \text{Pick 3 questions with the success rate closest to the mean.} \end{cases} $$

$$ \bullet n = 30 $$

$$ \bullet \theta = 66\% $$

<figure><img src="image_31.png" /><figcaption>(A) Reproducibility</figcaption></figure>

<figure><img src="image_32.png" /><figcaption>(B) Difficulty correlation</figcaption></figure>

<figure><img src="image_33.png" /><figcaption>(C) Complexity correlation</figcaption></figure>

<figure><img src="image_34.png" /><figcaption>(D) Aggregated metrics</figcaption></figure>

<caption>FIGURE 4.5: Metric results for different threshold approaches, aggregated (averaged) for different student groups. Figure 4.5d represents the aggregated (averaged) results of all 3 metrics, with tinted areas representing the standard deviation of results.</caption>

From the aggregated metric results in Figure 4.5d, we can notice that choosing the easiest questions leads to better performance on metrics. Indeed if one consults Table 4.5, choosing the easiest questions appears to bring the best outcomes with statistically significant differences from other approaches when measuring reproducibility and difficulty correlation. For complexity correlation choosing the easiest questions and the ones with the mean difficulty seems to perform comparably well.

The observed behavior can be explained by the fact that mastering the easiest questions appears to correlate the best with desired performance. Indeed, if a question linked to a certain skill is more difficult for students than other questions, one
might verify if this question tests any other skills. If yes, such a question might not be suitable as a response of $p(s)$.

Overall, based on our experiments we chose the following setup:

* $n = 30$

* $\mu(q) : Q_{easiest} \mapsto S_{finer}$

* $p(s) : S_{finer} \mapsto Q_{easiest}$

* $\theta = 66\%$

As can be observed from Figure 4.5, the used parameters of the system yield high reproducibility (in the range of [0.6, 0.8]), as well as high difficulty correlation (in the range of [0.6, 0.8] as well). However, the complexity correlation is relatively low (in the range of [0.3, 0.4]). This could be a result of our dataset, where the difficulty of questions does not correlate with high complexity<fnref n="3" />. However, it is necessary to go back to the experiment in Section 4.1 to see that before further parameter modifications, the observed complexity correlation was in the range [0, 0.3] signifying that not only parameter design can improve the metric results, but also make the performance of the model more stable.

<table><thead><tr><th></th><th>All questions</th><th>Average difficulty</th><th>Hardest</th><th>Easiest</th></tr></thead><tbody><tr><th>All questions</th><td>1.0</td><td>7.5 * 10<sup>-1</sup> (0.31)</td><td>2.2 * 10<sup>-2</sup> (2.347)</td><td>2.6 * 10<sup>-2</sup> (-2.282)</td></tr><tr><th>Average difficulty</th><td>7.5 * 10<sup>-1</sup> (-0.31)</td><td>1.0 (0)</td><td>5.1 * 10<sup>-2</sup> (1.99)</td><td>1.6 * 10<sup>-2</sup> (-2.485)</td></tr><tr><th>Hardest</th><td>2.2 * 10<sup>-2</sup> (-2.347)</td><td>5.1 * 10<sup>-2</sup> (-1.99)</td><td>1.0 (0)</td><td>4.7 * 10<sup>-5</sup> (-4.397)</td></tr><tr><th>Easiest</th><td>2.6 * 10<sup>-2</sup> (2.282)</td><td>1.6 * 10<sup>-2</sup> (2.485)</td><td>4.7 * 10<sup>-5</sup> (4.397)</td><td>1.0</td></tr></tbody></table>

<table><caption>(A) Reproducibility</caption><thead><tr><th></th><th>All questions</th><th>Average difficulty</th><th>Hardest</th><th>Easiest</th></tr></thead><tbody><tr><th>All questions</th><td>1.0 (0)</td><td>4 * 10<sup>-3</sup> (-2.984)</td><td>3.7 * 10<sup>-1</sup> (0.905)</td><td>9.9 * 10<sup>-7</sup> (-5.473)</td></tr><tr><th>Average difficulty</th><td>4 * 10<sup>-3</sup> (2.984)</td><td>1.0 (0)</td><td>3.3 * 10<sup>-4</sup> (3.812)</td><td>2.5 * 10<sup>-2</sup> (-2.293)</td></tr><tr><th>Hardest</th><td>3.7 * 10<sup>-1</sup> (-0.905)</td><td>3.3 * 10<sup>-4</sup> (-3.812)</td><td>1.0 (0)</td><td>4.3 * 10<sup>-8</sup> (-6.301)</td></tr><tr><th>Easiest</th><td>9.9 * 10<sup>-7</sup> (5.473)</td><td>2.5 * 10<sup>-2</sup> (2.293)</td><td>4.3 * 10<sup>-8</sup> (6.301)</td><td>1.0 (0)</td></tr></tbody></table>

<table><caption>(C) Complexity correlation</caption><caption>(B) Difficulty correlation</caption><thead><tr><th></th><th>All questions</th><th>Average difficulty</th><th>Hardest</th><th>Easiest</th></tr></thead><tbody><tr><th>All questions</th><td>1.0 (0)</td><td>5.4 * 10<sup>-5</sup> (-4.362)</td><td>9.7 * 10<sup>-1</sup> (-0.035)</td><td>1.6 * 10<sup>-4</sup> (-4.033)</td></tr><tr><th>Average difficulty</th><td>5.4 * 10<sup>-5</sup> (4.362)</td><td>1.0 (0)</td><td>5 * 10<sup>-4</sup> (3.688)</td><td>8.3 * 10<sup>-1</sup> (-0.212)</td></tr><tr><th>Hardest</th><td>9.7 * 10<sup>-1</sup> (0.035)</td><td>5 * 10<sup>-4</sup> (-3.688)</td><td>1.0 (0)</td><td>8.5 * 10<sup>-4</sup> (-3.519)</td></tr><tr><th>Easiest</th><td>1.6 * 10<sup>-4</sup> (4.033)</td><td>8.3 * 10<sup>-1</sup> (0.212)</td><td>8.5 * 10<sup>-4</sup> (3.519)</td><td>1.0 (0)</td></tr></tbody></table>

<caption>TABLE 4.5: T-test results (p-value and t-statistic) from comparing metric results for different question selection approaches. The p-value threshold is set to be $p - value \le 0.05$.</caption>

## 4.4 Comparing with the Baseline

All previous experiments involved building a knowledge space, converting it into a graph, and updating its weights for 30 iterations. As one can assume, this approach

<footnote n="3">³Find an example in Table 4.2. The correlation ( Spearman $\rho$) between difficulty and complexity rank on all data was observed to be 0.3.</footnote>
<table><caption>TABLE 4.6: P-values for t-test performed comparing metric results for all students using learning path from trained weights and based on conditional probabilities. The p-value threshold is set to be *p* - *value* ≤ 0.05.</caption><thead><tr><td>Metric</td><td>P-value</td></tr></thead><tbody><tr><td>Reproducibility</td><td>2.8 * 10<sup>-5</sup></td></tr><tr><td>Difficulty correlation</td><td>8.8 * 10<sup>-1</sup></td></tr><tr><td>Complexity correlation</td><td>9.1 * 10<sup>-1</sup></td></tr></tbody></table>

is time and memory-consuming (more on this in Section 6.1). The following question is expected: is this resource allocation justifiable?

For this experiment, we use the best-found parameters described in the section before and attempt to utilize the same parameters to create a learning path using conditional probabilities. As shown in Figure 4.6, the results of complexity and difficulty correlation are comparable when using both approaches, however, there is a drastic difference in the reproducibility of these types of paths. In fact, this can be confirmed with the t-test results seen in Table 4.6.

<figure><img src="image_35.png" /><figcaption>FIGURE 4.6: Metric results for building a path using trained weights and using conditional probabilities, aggregated (averaged) for different student groups. Figure 4.6d represents the aggregated (averaged) results of all 3 metrics, with tinted areas representing the standard deviation of results.</figcaption></figure>

Therefore, one might conclude that if the reproducibility of recommendations is of importance to the creators of an ITS system, it is best to attempt and learn the weights needed to traverse the knowledge space, otherwise, the produced result might not be well representative of unseen student groups. In case one decides
to adopt a lightweight approach for computing a learning path, we still propose
following the same attribute design experiments as were performed for the trained
path in this chapter.
# Chapter 5

# Discussion

After presenting all experiment results, it is worth going back to the original questions posed at the beginning of the project and presenting our answers.

**Given responses to a language proficiency test, can we create such an ITS that would recommend EFL teachers a learning path that is optimal for their students?**

It is indeed possible to create such a system that would produce learning path recommendations that can generalize well (obtain high reproducibility for different sets of students), and represent the data well (yield an elevated difficulty correlation), however, still perform relatively poorly on representing the expected complexity of questions and skills. One explanation for such results would be that the resources (CEFR levels and GSE scores) used to extract the expected complexity of questions in the study are more suited for students who have no experience with the language and therefore are not usable in our case. Another potential explanation is that designing an ITS via FCA + KST methodology creates a learning path that poorly correlates to complexity. However, at this stage, we can not answer surely why the produced system performed relatively poorly on the complexity correlation.

**How a learning path recommendation can be evaluated in an offline environment?**

As mentioned above, we introduced a group of metrics that served as a good tool to compare different setups, nevertheless, the correlation of this evaluation approach with the real-world performance of such recommendations needs to be further studied.

**Can we improve the performance of such a system by just changing parameters while using the same data?**

With the metrics we selected above, it is certainly possible to improve the system performance by just redefining what items and skills should be used by the system. During this procedure, neither new answers were obtained, nor new questions were added. Therefore, we want to highlight the importance of processing the data correctly before utilizing it for an ITS creation.

In our research, using a mastery threshold that allows for a careless error when students are answering questions, and only selecting the easiest questions to represent a skill, appeared to lead to the best outcomes. Nonetheless, additional studies are required to validate if these parameters are specific to our given dataset or indicate a more global trend. However, we strongly suggest the same type of experiments performed in this study to find the best-performing attribute definitions before implementing an ITS.
<figure><img src="image_36.png" /></figure>
# Chapter 6

## Conclusion

This project was kickstarted with the objective of creating a system that would recommend to teachers a sequence of topics to cover in their class while preparing for a TOEIC exam. An ideal learning path would be applicable to different student groups, represent the observed difficulty of questions, and encode the perceived complexity of the subject by experts.

In order to tackle this task we turned to ITS, that are prevalent in the field of language learning. Out of the multitude of approaches to developing such a system, we turned to implement an integration of KST and FCA, since it allows for the automatic mining of all possible knowledge states from observed answers of students, thus not having to model or train any hidden states. During system development, we tested different configurations and proposed a methodology to evaluate them, according to the originally set objective.

In this chapter, we will provide the final words about the project, covering the limitations of the study in Section 6.1, Section 6.2 will discuss how these limitations could be addressed in further research, as well as pose some additional questions for further researchers, and finally Section 6.3 will provide the summary of main findings.

### 6.1 Limitations

Due to time and resource restrictions associated with the project, there were multiple limitations when implementing this project that require further discussion.

**Lack of qualitative evaluation** Because of the time constraints, the original plan to utilize a qualitative evaluation was scrapped. The idea behind this approach lies in querying experts in the domain of teaching English and getting their evaluation of the recommended learning paths. Since this option was no longer available, it was decided to utilize CEFR competencies as a way of assessing experts' opinions. However, due to lack of feedback, it is hard to understand whether the produced system really underperforms on qualitative evaluation, or whether CEFR evaluation is simply unsuitable for the task.

**Only offline evaluation** While a lot of studies related to creating learning path recommendations are not evaluated in real-world scenarios (Silva et al., 2023), this research could still benefit from applying the recommended learning paths in classrooms and comparing student results with students who followed the curriculum proposed by a teacher.
<figure><img src="image_37.png" /><figcaption>FIGURE 6.1: Dependency of runtime from the size of knowledge space. The regression line drawn at $y = \frac{x}{47}$.</figcaption></figure>

**Naive algorithms** Since the goal of the project was to create an ITS, little attention was given to the optimization of applied algorithms. Due to the relatively small size of the used dataset, the computational complexity did not pose a problem. All code was run locally on a MacBook Air with M1 chip and 8GB RAM and a run of 2-fold with 30 repetitions took 20 minutes or less.

As can be seen in Figure 6.1, even though most of the computations performed in the scope of the work showed a close to linear dependency from the size of the knowledge space, as the knowledge spaces grow bigger, the computation times start to explode. One of the ways to address this would be to use a sub-profile of students during computation, since as one can observe much fewer unique knowledge states were found for student sub-groups, hence there is a lower chance of computation time exploding. However, if there is a goal to introduce more attributes, the algorithms need to be adjusted.

**Effect of Repetition Number** As was mentioned above all experiments were set up as 2-fold splits with 30 repetitions, however, the impact of the number of repetitions is kept outside of the scope of this work. For further steps, it is important to consider setups with different number of iterations.

## 6.2 Future Work

Firstly, in future work, we would like to address all the limitations mentioned in the previous section. Additionally, we would like to propose a series of topics that we believe are important for further research in the field.

### 6.2.1 Granularity of Attributes

As shown in some examples from Section 2.2, fine-grained attributes can stand for topics of different depths. We highlight the necessity of the research regarding different granularities. Frequently (as done in the experiments above) the desired granularity is dictated by an expert opinion, however, at this stage, there is no methodology for soundly proving that a selected granularity is appropriate for the task. A question might be posed: is there a method for standardizing the granularity definitions? If yes, is there a way to identify what granularity is the most desirable for an ITS design?
6.2.2 Skill Map and Question Design

To the best of our knowledge, the effect of item design on an ITS performance is not well studied. In this study, the set of questions was manually mapped to a set of skills. Another approach would be to keep the same set of attributes and construct new questions, tailor-made for the ITS. We propose the following topic for discussion: how the question design affects the system performance?

Above, we hypothesized that picking the easiest questions might be a desirable approach to test students' knowledge of a topic, however, if the same "easy" question is phrased differently, will we yield the same results?

6.2.3 Unconscious Biases in the ITS Design

Both discussion topics proposed above talk about parts of an ITS development that heavily rely on human expertise, which can lead to the undesirable introduction of biases. Even though research in this field is present (Shalini and Tewari, 2020; Lin, Huang, and Lu, 2023), with the widespread of ITS in the education field it is crucial to prioritize this research area.

6.2.4 Impact of Student Profiles

Even though we introduced student profiling in the scope of this work, it was not rigorously investigated. Additionally, to the language levels, we believe it is important to consider the demographic information of the students. If the system created for French-speaking English learners is applied to Russian-speaking English learners, will it hold the same results? For this reason, it is necessary to continue the research on the impact of student profiles on the created systems (which could be interconnected with the study of biases).

**6.3 Main Findings**

If we are to summarize our main conclusions for the reader, we want to highlight
the following:

1. If the original data does not correlate well with a target metric (in our case complexity extracted via GSE scores), the produced ITS will likely not be well correlated with the metric as well.

2. Despite that, it is still possible to improve the ITS performance on the target metric using careful item and attribute selection, therefore, this step is strongly recommended for designers of such systems.

3. One of the strongest metric improvements observed in the scope of this pa-
per was related to only picking the easiest questions to verify mastery of an
attribute. We hypothesize that this is indeed true since harder questions can
potentially involve other skills that went unnoticed by experts. Nevertheless,
the hypothesis needs to be confirmed via further research.

4. Despite naive approaches towards the creation of surmise relations and travers-
ing knowledge spaces, creating a knowledge space with an FCA integration is
feasible on a local machine and does not require additional computing power,
which makes it a powerful tool.
We hope that the findings and the experiments performed in the scope of this
project can serve as a solid foundation for the subsequent development of an ITS by
Naimrod.
# Appendix A

## Attributes

### A.1 Attribute Distribution

In this section, we will introduce some distributions regarding attributes: number of questions per attribute, attributes' success rates, and corresponding CEFR and GSE labels.

#### A.1.1 Coarse Attributes

<figure><img src="image_38.png" /><figcaption>FIGURE A.1: Number of questions associated with an attribute (coarse attributes).</figcaption></figure>
<figure><img src="image_39.png" /><figcaption>FIGURE A.2: Success rates per coarse attribute for different student profiles (all, beginners, intermediate, advanced).</figcaption></figure>

<table><caption>TABLE A.1: Coarse attributes, their corresponding average GSE scores, CEFR levels, and success rates (SR) for different groups of students.</caption><thead><tr><th></th><th>Attribute</th><th>GSE</th><th>CEFR</th><th>SR (All)</th><th>SR (Beginners)</th><th>SR (Intermediate)</th><th>SR (Advanced)</th></tr></thead><tbody><tr><td>1</td><td>Conjunction.And</td><td>27.5</td><td>A1</td><td>69.15%</td><td>43.48%</td><td>76.19%</td><td>93.75%</td></tr><tr><td>2</td><td>Prep Direction Location Target</td><td>30.3</td><td>A2.1</td><td>65.51%</td><td>44.44%</td><td>65.48%</td><td>95.83%</td></tr><tr><td>3</td><td>Adverb.Time</td><td>35.0</td><td>A2.1</td><td>71.94%</td><td>46.96%</td><td>80.00%</td><td>93.75%</td></tr><tr><td>4</td><td>Position.Adverb</td><td>36.0</td><td>A2.2</td><td>68.66%</td><td>43.48%</td><td>77.86%</td><td>88.75%</td></tr><tr><td>5</td><td>ImperativeClause</td><td>36.3</td><td>A2.2</td><td>67.16%</td><td>56.52%</td><td>64.29%</td><td>87.50%</td></tr><tr><td>6</td><td>Prep.Time</td><td>37.8</td><td>A2.2</td><td>72.95%</td><td>52.72%</td><td>74.55%</td><td>99.22%</td></tr><tr><td>7</td><td>Tense.Present</td><td>39.2</td><td>A2.2</td><td>67.00%</td><td>44.93%</td><td>73.81%</td><td>86.81%</td></tr><tr><td>8</td><td>Adjective.Comparison</td><td>40.7</td><td>A2.2</td><td>76.12%</td><td>56.52%</td><td>83.33%</td><td>91.67%</td></tr><tr><td>9</td><td>Verb.Agreement</td><td>44.5</td><td>B1.1</td><td>56.16%</td><td>36.14%</td><td>58.93%</td><td>80.08%</td></tr><tr><td>10</td><td>Adverb.Meaning</td><td>45.0</td><td>B1.1</td><td>70.90%</td><td>47.83%</td><td>75.00%</td><td>96.88%</td></tr><tr><td>11</td><td>SimilarWords.Spelling</td><td>45.0</td><td>B1.1</td><td>61.79%</td><td>46.96%</td><td>57.14%</td><td>91.25%</td></tr><tr><td>12</td><td>Tense.Past</td><td>45.5</td><td>B1.1</td><td>55.22%</td><td>31.88%</td><td>57.14%</td><td>85.42%</td></tr><tr><td>13</td><td>Suffix.Noun</td><td>46.0</td><td>B1.1</td><td>74.13%</td><td>55.80%</td><td>76.19%</td><td>96.88%</td></tr><tr><td>14</td><td>Suffix.Adjective</td><td>47.0</td><td>B1.1</td><td>63.78%</td><td>44.06%</td><td>65.24%</td><td>89.58%</td></tr><tr><td>15</td><td>Tense.Agreement</td><td>47.8</td><td>B1.1</td><td>63.81%</td><td>42.39%</td><td>68.15%</td><td>86.98%</td></tr><tr><td>16</td><td>Conjunction.Time</td><td>48.0</td><td>B1.1</td><td>72.76%</td><td>45.65%</td><td>82.14%</td><td>95.31%</td></tr><tr><td>17</td><td>Suffix.Verb</td><td>50.0</td><td>B1.1</td><td>70.90%</td><td>45.65%</td><td>79.46%</td><td>92.19%</td></tr><tr><td>18</td><td>Conjunction.Negation</td><td>50.0</td><td>B1.1</td><td>49.75%</td><td>40.58%</td><td>51.19%</td><td>60.42%</td></tr><tr><td>19</td><td>Conjunction.Cause</td><td>50.3</td><td>B1.2</td><td>73.13%</td><td>56.52%</td><td>75.00%</td><td>93.75%</td></tr><tr><td>20</td><td>Conjunction.Contrast</td><td>50.3</td><td>B1.2</td><td>62.69%</td><td>37.68%</td><td>69.64%</td><td>86.46%</td></tr><tr><td>21</td><td>RelativeClause</td><td>50.7</td><td>B1.2</td><td>73.88%</td><td>47.83%</td><td>83.93%</td><td>93.75%</td></tr><tr><td>22</td><td>TenseConditional</td><td>53.0</td><td>B1.2</td><td>62.24%</td><td>40.00%</td><td>64.64%</td><td>90.00%</td></tr><tr><td>23</td><td>PassiveVoice</td><td>56.8</td><td>B1.2</td><td>56.42%</td><td>34.78%</td><td>57.14%</td><td>86.25%</td></tr><tr><td>24</td><td>Future</td><td>60.8</td><td>B2.1</td><td>62.19%</td><td>36.23%</td><td>69.05%</td><td>87.50%</td></tr></tbody></table>
A.1.2 Finer Attributes

<figure><img src="image_40.png" /><figcaption>FIGURE A.3: Number of questions associated with an attribute (finer attributes).</figcaption></figure>

<figure><img src="image_41.png" /><figcaption>FIGURE A.4: Success rates per finer attribute for different student profiles (all, beginners, intermediate, advanced).</figcaption></figure>
<table><caption>TABLE A.2: Finer attributes, their corresponding average GSE scores, CEFR levels, and success rates (SR) for different groups of students.</caption><thead><tr><th></th><th>Attribute</th><th>GSE</th><th>CEFR</th><th>SR (All)</th><th>SR (Beginners)</th><th>SR (Intermediate)</th><th>SR (Advanced)</th></tr></thead><tbody><tr><td>1</td><td>SubjectVerbAgreement</td><td>24.0</td><td>A1</td><td>65.37%</td><td>44.35%</td><td>70.71%</td><td>86.25%</td></tr><tr><td>2</td><td>PrepPlace.Basic</td><td>26.0</td><td>A1</td><td>66.17%</td><td>44.93%</td><td>66.67%</td><td>95.83%</td></tr><tr><td>3</td><td>Conjunction.And</td><td>27.5</td><td>A1</td><td>69.15%</td><td>43.48%</td><td>76.19%</td><td>93.75%</td></tr><tr><td>4</td><td>PrepPlace.Advanced</td><td>30.0</td><td>A2.1</td><td>59.20%</td><td>36.23%</td><td>58.33%</td><td>93.75%</td></tr><tr><td>5</td><td>PresentSimple</td><td>30.8</td><td>A2.1</td><td>63.54%</td><td>40.99%</td><td>69.90%</td><td>84.82%</td></tr><tr><td>6</td><td>PastSimple</td><td>32.2</td><td>A2.1</td><td>53.36%</td><td>28.26%</td><td>55.36%</td><td>85.94%</td></tr><tr><td>7</td><td>Adverb.Time</td><td>35.0</td><td>A2.1</td><td>71.94%</td><td>46.96%</td><td>80.00%</td><td>93.75%</td></tr><tr><td>8</td><td>Position.Adverb</td><td>36.0</td><td>A2.2</td><td>68.66%</td><td>43.48%</td><td>77.86%</td><td>88.75%</td></tr><tr><td>9</td><td>ImperativeClause</td><td>36.3</td><td>A2.2</td><td>67.16%</td><td>56.52%</td><td>64.29%</td><td>87.50%</td></tr><tr><td>10</td><td>Prep.Time</td><td>37.8</td><td>A2.2</td><td>72.71%</td><td>50.31%</td><td>76.02%</td><td>99.11%</td></tr><tr><td>11</td><td>PastParticiple</td><td>38.0</td><td>A2.2</td><td>69.25%</td><td>41.74%</td><td>78.57%</td><td>92.50%</td></tr><tr><td>12</td><td>PresentParticiple</td><td>38.0</td><td>A2.2</td><td>60.20%</td><td>43.48%</td><td>58.33%</td><td>87.50%</td></tr><tr><td>13</td><td>Adjective.Comparison</td><td>40.7</td><td>A2.2</td><td>76.12%</td><td>56.52%</td><td>83.33%</td><td>91.67%</td></tr><tr><td>14</td><td>SuffixAdjective.Basic</td><td>42.0</td><td>A2.2</td><td>61.94%</td><td>47.83%</td><td>62.50%</td><td>81.25%</td></tr><tr><td>15</td><td>PresentPerfect</td><td>43.9</td><td>B1.1</td><td>79.10%</td><td>58.70%</td><td>87.50%</td><td>93.75%</td></tr><tr><td>16</td><td>VerbAgreement.Modal</td><td>44.0</td><td>B1.1</td><td>50.75%</td><td>26.09%</td><td>48.21%</td><td>90.62%</td></tr><tr><td>17</td><td>Adverb.Meaning</td><td>45.0</td><td>B1.1</td><td>70.90%</td><td>47.83%</td><td>75.00%</td><td>96.88%</td></tr><tr><td>18</td><td>Prep.To</td><td>45.0</td><td>B1.1</td><td>73.88%</td><td>52.17%</td><td>76.79%</td><td>100.00%</td></tr><tr><td>19</td><td>Suffix.Noun</td><td>46.0</td><td>B1.1</td><td>74.13%</td><td>55.80%</td><td>76.19%</td><td>96.88%</td></tr><tr><td>20</td><td>Conditional.1</td><td>46.0</td><td>B1.1</td><td>65.97%</td><td>46.09%</td><td>67.14%</td><td>92.50%</td></tr><tr><td>21</td><td>SuffixAdjective.Advanced</td><td>48.0</td><td>B1.1</td><td>66.87%</td><td>46.96%</td><td>67.14%</td><td>95.00%</td></tr><tr><td>22</td><td>Conjunction.Time</td><td>48.0</td><td>B1.1</td><td>73.13%</td><td>50.43%</td><td>78.57%</td><td>96.25%</td></tr><tr><td>23</td><td>Prep.Between</td><td>49.0</td><td>B1.1</td><td>65.67%</td><td>52.17%</td><td>60.71%</td><td>93.75%</td></tr><tr><td>24</td><td>Conjunction.Negation</td><td>50.0</td><td>B1.1</td><td>49.75%</td><td>40.58%</td><td>51.19%</td><td>60.42%</td></tr><tr><td>25</td><td>Suffix.Verb</td><td>50.0</td><td>B1.1</td><td>65.67%</td><td>39.13%</td><td>71.43%</td><td>93.75%</td></tr><tr><td>26</td><td>Conjunction.Cause</td><td>50.3</td><td>B1.2</td><td>73.13%</td><td>56.52%</td><td>75.00%</td><td>93.75%</td></tr><tr><td>27</td><td>VerbAgreement</td><td>50.3</td><td>B1.2</td><td>60.60%</td><td>44.35%</td><td>63.57%</td><td>78.75%</td></tr><tr><td>28</td><td>Conjunction.Contrast</td><td>50.3</td><td>B1.2</td><td>62.69%</td><td>37.68%</td><td>69.64%</td><td>86.46%</td></tr><tr><td>29</td><td>RelativeClause</td><td>50.7</td><td>B1.2</td><td>73.88%</td><td>47.83%</td><td>83.93%</td><td>93.75%</td></tr><tr><td>30</td><td>VerbAgreement.Subjunctive</td><td>53.0</td><td>B1.2</td><td>58.71%</td><td>39.13%</td><td>65.48%</td><td>75.00%</td></tr><tr><td>31</td><td>VerbAgreement.Causative</td><td>56.0</td><td>B1.2</td><td>35.82%</td><td>13.04%</td><td>35.71%</td><td>68.75%</td></tr><tr><td>32</td><td>PassiveVoice</td><td>56.8</td><td>B1.2</td><td>53.73%</td><td>33.70%</td><td>51.79%</td><td>85.94%</td></tr><tr><td>33</td><td>Conditional.23</td><td>57.7</td><td>B1.2</td><td>58.51%</td><td>33.91%</td><td>62.14%</td><td>87.50%</td></tr><tr><td>34</td><td>PastPerfect</td><td>60.8</td><td>B2.1</td><td>58.96%</td><td>39.13%</td><td>60.71%</td><td>84.38%</td></tr><tr><td>35</td><td>Future</td><td>60.8</td><td>B2.1</td><td>64.18%</td><td>39.13%</td><td>73.21%</td><td>84.38%</td></tr></tbody></table>
## A.2 Question Examples

In this section, we will provide some examples of questions and the connected skills (coarse and finer).

### A.2.1 CEFR A1-A2 Attributes

<table><caption>TABLE A.3: Attributes s with CEFR level A1-A2.2 and an example of q ∈ Qall returned by the problem function p(s). Note that multiple skills can be required to solve a question.</caption><thead><tr><th></th><th>Finer Attribute</th><th>Coarse Attribute</th><th>Question</th><th>Answers</th></tr></thead><tbody><tr><td>1</td><td>SubjectVerbAgreement</td><td>VerbAgreement</td><td>If the customer __ not satisfied, please have him call the manager.</td><td>A) am<br>B) is<br>C) are<br>D) be</td></tr><tr><td>2</td><td>PrepPlace.Basic</td><td>Prep tap</td><td>A letter for Mr. Carn was left __ Mr. Britto's desk.</td><td>A) through<br>B) on<br>C) out<br>D) up</td></tr><tr><td>3</td><td>Conjunction.And</td><td>Conjunction.And</td><td>Our company stands for quality __ design.</td><td>A) or<br>B) and<br>C) but<br>D) neither</td></tr><tr><td>4</td><td>PrepPlace.Advanced</td><td>Prep tap</td><td>The purchasing department is located __ the reception desk.</td><td>A) across<br>B) between<br>C) behind<br>D) from</td></tr><tr><td>5</td><td>PresentSimple</td><td>Tense.Present</td><td>The last train to Hamburg __ at 10.30.</td><td>A) depart<br>B) departs<br>C) to depart<br>D) departing</td></tr><tr><td>6</td><td>PastSimple</td><td>Tense.Past</td><td>The head of sales __ to San Diego for the annual sales convention.</td><td>A) went<br>B) gone<br>C) go<br>D) going</td></tr><tr><td>7</td><td>Adverb.Time</td><td>Adverb.Time</td><td>Have you ever used this type of copy machine __?</td><td>A) berore<br>B) prior<br>C) previous<br>D) advance</td></tr><tr><td>8</td><td>Position.Adverb</td><td>Position.Adverb</td><td>A manager __ new things from his or her staff.</td><td>A) can<br>sometimes<br>learn<br>B) learning<br>sometimes<br>can<br>C) sometimes<br>can<br>learning<br>D) sometimes<br>learn<br>can</td></tr><tr><td>9</td><td>ImperativeClause</td><td>ImperativeClause</td><td>When you need supplies, __ a request with the office manager.</td><td>A) filing<br>B) fell<br>C) fallen<br>D) file</td></tr><tr><td>10</td><td>Prep.Time</td><td>Prep.Time</td><td>The finance committee will meet again __ the 8th of May.</td><td>A) for<br>B) to<br>C) from<br>D) on</td></tr><tr><td>11</td><td>PastParticiple</td><td>Suffix.Adjective</td><td>Mr. Hatori was very __ when he got a promotion.</td><td>A) excite<br>B) exciting<br>C) excited<br>D) excites</td></tr><tr><td>12</td><td>PresentParticiple</td><td>Suffix.Adjective</td><td>Transfering our __ facilities to areas with low labor costs will lead to greater profitability in the long term.</td><td>A) manufactured<br>B) manufactures<br>C) manufacturing<br>D) manufacturer</td></tr><tr><td>13</td><td>Adjective.Comparison</td><td>Adjective.Comparison</td><td>This list of contributors is more __ that one.</td><td>A) current<br>B) currently<br>C) current than<br>D) current as</td></tr><tr><td>14</td><td>SuffixAdjective.Basic</td><td>Suffix.Adjective</td><td>These addresses should be listed in __ order.</td><td>A) alphabet<br>B) alphabetize<br>C) alphabetically<br>D) alphabetical</td></tr></tbody></table>
A.2.2 CEFR B1.1 Attributes

<table><caption>TABLE A.4: Attributes *s* with CEFR level B1.1 and an example of *q* ∈ Q<sub>all</sub> returned by the problem function *p*(*s*). Note that multiple skills can be required to solve a question.</caption><thead><tr><th></th><th>Finer Attribute</th><th>Coarse Attribute</th><th>Question</th><th>Answers</th></tr></thead><tbody><tr><td>15</td><td>PresentPerfect</td><td>Tense.Present</td><td>Mr. Ni ___ in charge of research for two years.</td><td>A) has<br>B) has been<br>C) is<br>D) had</td></tr><tr><td>16</td><td>VerbAgreement.Modal</td><td>Verb.Agreement</td><td>The guard must ___ your identification at the gate.</td><td>A) checked<br>B) checking<br>C) checks<br>D) check</td></tr><tr><td>17</td><td>Adverb.Meaning</td><td>Adverb.Meaning</td><td>Because his finger was broken, the secretary could not type ___.</td><td>A) efficiently<br>B) intermittently<br>C) slowly<br>D) gradually</td></tr><tr><td>18</td><td>Prep.To</td><td>Prep DirectionLocationTarget</td><td>The training session has been changed from 8.30 ___ 9.00.</td><td>A) at<br>B) to<br>C) in<br>D) by</td></tr><tr><td>19</td><td>Suffix.Noun</td><td>Suffix.Noun</td><td>Our goal is to turn ___ into success.</td><td>A) failing<br>B) fail<br>C) failed<br>D) failure</td></tr><tr><td>20</td><td>Conditional.1</td><td>Tense Conditional</td><td>If the waiter can not handle your request, the captain ___ assist you.</td><td>A) will<br>B) has<br>C) did<br>D) is</td></tr><tr><td>21</td><td>SuffixAdjective.Advanced</td><td>Suffix.Adjective</td><td>Is the annual report ___ yet?</td><td>A) avail<br>B) available<br>C) availability<br>D) availing</td></tr><tr><td>22</td><td>Conjunction.Time</td><td>Conjunction.Time</td><td>___ the meeting, Ms. Tran missed important phone calls.</td><td>A) Although<br>B) In spite of<br>C) Because<br>D) During</td></tr><tr><td>23</td><td>Prep.Between</td><td>Prep DirectionLocationTarget</td><td>The fax machine is ___ the postage meter and the copy machine.</td><td>A) among<br>B) outside<br>C) between<br>D) through</td></tr><tr><td>24</td><td>Conjunction.Negation</td><td>Conjunction.Negation</td><td>The billing clerk was not able to find the invoice ___ the order.</td><td>A) or<br>B) and<br>C) but<br>D) though</td></tr><tr><td>25</td><td>Suffix.Verb</td><td>Suffix.Verb</td><td>Ms. Najar wants to ___ the costs tonight.</td><td>A) final<br>B) finalize<br>C) finally<br>D) finality</td></tr></tbody></table>
A.2.3 CEFR B1.2-B2.1 Attributes

<table><caption>TABLE A.5: Attributes s with CEFR level B1.1-B2.1 and an example of q ∈ Qall returned by the problem function p(s). Note that multiple skills can be required to solve a question.</caption><thead><tr><th></th><th>Finer Attribute</th><th>Coarse Attribute</th><th>Question</th><th>Answers</th></tr></thead><tbody><tr><td>26</td><td>Conjunction.Cause</td><td>Conjunction.Cause</td><td>The meeting is postponed Mr. Tan's plane was late.</td><td>A) although<br>B) while<br>C) because<br>D) with</td></tr><tr><td>27</td><td>VerbAgreement</td><td>VerbAgreement</td><td>The visitors are reminded name tags at all times.</td><td>A) to wear<br>B) wear<br>C) be worn<br>D) is wearing</td></tr><tr><td>28</td><td>Conjunction.Contrast</td><td>Conjunction.Contrast</td><td>The restaurant is open on weekends not on holidays.</td><td>A) either<br>B) or<br>C) so<br>D) but</td></tr><tr><td>29</td><td>RelativeClause</td><td>RelativeClause</td><td>The person lost a briefcase may claim it in the lobby.</td><td>A) whose<br>B) which<br>C) whom<br>D) who</td></tr><tr><td>30</td><td>VerbAgreement.Subjunctive</td><td>VerbAgreement</td><td>The bell captain suggested that more porters hired.</td><td>A) are<br>B) have<br>C) be<br>D) do</td></tr><tr><td>31</td><td>VerbAgreement.Causative</td><td>VerbAgreement</td><td>The trainers for the seminar had the crew their equipment to the conference center.</td><td>A) move<br>B) moving<br>C) mover<br>D) moved</td></tr><tr><td>32</td><td>PassiveVoice</td><td>PassiveVoice</td><td>The seminar was cancelled because the invitations were not on time.</td><td>A) printer<br>B) printing<br>C) printed<br>D) print</td></tr><tr><td>33</td><td>Conditional.23</td><td>Tense.Conditional</td><td>If we had started, we the deadline.</td><td>A) would meet<br>B) would have met<br>C) will meet<br>D) will have meet</td></tr><tr><td>34</td><td>PastPerfect</td><td>Tense.Past</td><td>The final purchase price was higher than the investors.</td><td>A) had expected<br>B) expect<br>C) are expecting<br>D) will expect</td></tr><tr><td>35</td><td>Future</td><td>Tense.Future</td><td>Employees who attending the conference can get a discount on travel arrangements.</td><td>A) have going<br>B) are going<br>C) will<br>D) will be</td></tr></tbody></table>
<figure><img src="image_42.png" /></figure>
# Appendix B

## Learning Paths

In this Appendix, we will present the learning paths produced as the result of the training weights process with the following training parameters:

* $n = 30$

* $\mu(q) : Q_{easiest} \mapsto S_{finer}$

* $p(s) : S_{finer} \mapsto Q_{easiest}$

* $\theta = 66\%$

In the tables below you will find skills, step they were recommended at, corresponding CEFR and GSE labels, and the success rate<fnref n="1" />.

### B.1 All Students

<table><caption>TABLE B.1: The proposed learning path for all students. NB: Several skills can be proposed at the same step of the learning path.</caption><thead><tr><th>Step</th><th>Attribute</th><th>CEFR</th><th>GSE</th><th>Success Rate</th></tr></thead><tbody><tr><td>1</td><td>Prep.Time</td><td>A2.2</td><td>37.8</td><td>72.71%</td></tr><tr><td>2</td><td>Conjunction.Cause</td><td>B1.1</td><td>50.3</td><td>73.13%</td></tr><tr><td>3</td><td>Conjunction.Contrast</td><td>B1.1</td><td>50.3</td><td>62.69%</td></tr><tr><td>4</td><td>PresentSimple</td><td>A2.1</td><td>30.8</td><td>60.70%</td></tr><tr><td>5</td><td>PastParticiple</td><td>A2.2</td><td>38.0</td><td>69.25%</td></tr><tr><td>6</td><td>SubjectVerbAgreement</td><td>A1</td><td>24.0</td><td>65.37%</td></tr><tr><td>7</td><td>PrepPlace.Basic</td><td>A1</td><td>26.0</td><td>66.17%</td></tr><tr><td>8</td><td>Conjunction.And</td><td>A1</td><td>27.5</td><td>69.15%</td></tr><tr><td>9</td><td>VerbAgreement.Subjunctive</td><td>B1.2</td><td>53.0</td><td>58.71%</td></tr><tr><td>10</td><td>PresentPerfect</td><td>B1.1</td><td>43.9</td><td>73.13%</td></tr><tr><td>11</td><td>SuffixAdjective.Advanced</td><td>B1.1</td><td>48.0</td><td>66.87%</td></tr><tr><td>11</td><td>Conditional.1</td><td>B1.1</td><td>46.0</td><td>68.16%</td></tr><tr><td>12</td><td>PrepPlace.Advanced</td><td>A2.1</td><td>30.0</td><td>59.20%</td></tr><tr><td>12</td><td>Prep.To</td><td>B1.1</td><td>45.0</td><td>73.88%</td></tr><tr><td>12</td><td>Prep.Between</td><td>B1.1</td><td>49.0</td><td>65.67%</td></tr><tr><td>12</td><td>PastSimple</td><td>A2.1</td><td>32.2</td><td>53.36%</td></tr><tr><td>13</td><td>SuffixAdjective.Basic</td><td>A2.2</td><td>42.0</td><td>61.94%</td></tr><tr><td>14</td><td>Adjective.Comparison</td><td>A2.2</td><td>40.7</td><td>76.12%</td></tr><tr><td>15</td><td>PresentParticiple</td><td>A2.2</td><td>38.0</td><td>61.19%</td></tr><tr><td>15</td><td>Conditional.23</td><td>B1.2</td><td>57.7</td><td>58.51%</td></tr><tr><td>15</td><td>VerbAgreement</td><td>B1.1</td><td>50.3</td><td>60.60%</td></tr><tr><td>16</td><td>ImperativeClause</td><td>A2.2</td><td>36.3</td><td>67.16%</td></tr><tr><td>17</td><td>PassiveVoice</td><td>B1.2</td><td>56.8</td><td>63.43%</td></tr><tr><td>18</td><td>Adverb.Meaning</td><td>B1.1</td><td>45.0</td><td>70.90%</td></tr><tr><td>19</td><td>PastPerfect</td><td>B2.1</td><td>60.8</td><td>58.96%</td></tr><tr><td>19</td><td>VerbAgreement.Causative</td><td>B1.2</td><td>56.0</td><td>35.82%</td></tr></tbody></table>

<footnote n="1">¹The success rate for all questions, not only the easiest questions.</footnote>
## B.2 Beginners

<table><caption>TABLE B.2: The proposed learning path for beginners. NB: Several skills can be proposed at the same step of the learning path.</caption><thead><tr><th>Step</th><th>Attribute</th><th>CEFR</th><th>GSE</th><th>Success Rate</th></tr></thead><tbody><tr><td>1</td><td>Conjunction.Cause</td><td>B1.1</td><td>50.3</td><td>56.52%</td></tr><tr><td>2</td><td>Prep.Time</td><td>A2.2</td><td>37.8</td><td>50.31%</td></tr><tr><td>3</td><td>PastParticiple</td><td>A2.2</td><td>38.0</td><td>41.74%</td></tr><tr><td>4</td><td>PrepPlace.Basic</td><td>A1</td><td>26.0</td><td>44.93%</td></tr><tr><td>5</td><td>VerbAgreement.Subjunctive</td><td>B1.2</td><td>53.0</td><td>39.13%</td></tr><tr><td>5</td><td>SuffixAdjective.Advanced</td><td>B1.1</td><td>48.0</td><td>46.96%</td></tr><tr><td>5</td><td>PresentPerfect</td><td>B1.1</td><td>43.9</td><td>52.17%</td></tr><tr><td>6</td><td>Conditional.23</td><td>B1.2</td><td>57.7</td><td>33.91%</td></tr><tr><td>7</td><td>Conjunction.And</td><td>A1</td><td>27.5</td><td>43.48%</td></tr><tr><td>8</td><td>VerbAgreement</td><td>B1.1</td><td>50.3</td><td>44.35%</td></tr><tr><td>8</td><td>SuffixAdjective.Basic</td><td>A2.2</td><td>42.0</td><td>47.83%</td></tr><tr><td>8</td><td>SubjectVerbAgreement</td><td>A1</td><td>24.0</td><td>44.35%</td></tr><tr><td>8</td><td>PresentSimple</td><td>A2.1</td><td>30.8</td><td>37.68%</td></tr><tr><td>8</td><td>PresentParticiple</td><td>A2.2</td><td>38.0</td><td>56.52%</td></tr><tr><td>8</td><td>PrepPlace.Advanced</td><td>A2.1</td><td>30.0</td><td>36.23%</td></tr><tr><td>8</td><td>Adjective.Comparison</td><td>A2.2</td><td>40.7</td><td>56.52%</td></tr><tr><td>8</td><td>VerbAgreement.Causative</td><td>B1.2</td><td>56.0</td><td>13.04%</td></tr><tr><td>8</td><td>PastSimple</td><td>A2.1</td><td>32.2</td><td>28.26%</td></tr><tr><td>8</td><td>PastPerfect</td><td>B2.1</td><td>60.8</td><td>39.13%</td></tr><tr><td>8</td><td>PassiveVoice</td><td>B1.2</td><td>56.8</td><td>45.65%</td></tr><tr><td>8</td><td>ImperativeClause</td><td>A2.2</td><td>36.3</td><td>56.52%</td></tr><tr><td>8</td><td>Conjunction.Contrast</td><td>B1.1</td><td>50.3</td><td>37.68%</td></tr><tr><td>8</td><td>Conditional.1</td><td>B1.1</td><td>46.0</td><td>50.72%</td></tr><tr><td>8</td><td>Adverb.Meaning</td><td>B1.1</td><td>45.0</td><td>47.83%</td></tr><tr><td>8</td><td>Prep.To</td><td>B1.1</td><td>45.0</td><td>52.17%</td></tr><tr><td>8</td><td>Prep.Between</td><td>B1.1</td><td>49.0</td><td>52.17%</td></tr></tbody></table>
B.3 Intermediate

<table><caption>TABLE B.3: The proposed learning path for intermediate students. NB: Several skills can be proposed at the same step of the learning path. Step 0 stands for knowledge mastered by all intermediate students.</caption><thead><tr><th>Step</th><th>Attribute</th><th>CEFR</th><th>GSE</th><th>Success Rate</th></tr></thead><tbody><tr><td>0</td><td>Prep.Time</td><td>A2.2</td><td>37.8</td><td>76.02%</td></tr><tr><td>1</td><td>PresentSimple</td><td>A2.1</td><td>30.8</td><td>66.07%</td></tr><tr><td>2</td><td>SubjectVerbAgreement</td><td>A1</td><td>24.0</td><td>70.71%</td></tr><tr><td>3</td><td>PastParticiple</td><td>A2.2</td><td>38.0</td><td>78.57%</td></tr><tr><td>4</td><td>Conjunction.And</td><td>A1</td><td>27.5</td><td>76.19%</td></tr><tr><td>5</td><td>Conjunction.Cause</td><td>B1.1</td><td>50.3</td><td>75.00%</td></tr><tr><td>6</td><td>Conjunction.Contrast</td><td>B1.1</td><td>50.3</td><td>69.64%</td></tr><tr><td>7</td><td>VerbAgreement.Subjunctive</td><td>B1.2</td><td>53.0</td><td>65.48%</td></tr><tr><td>8</td><td>Conditional.23</td><td>B1.2</td><td>57.7</td><td>62.14%</td></tr><tr><td>9</td><td>VerbAgreement</td><td>B1.1</td><td>50.3</td><td>63.57%</td></tr><tr><td>10</td><td>PresentPerfect</td><td>B1.1</td><td>43.9</td><td>78.57%</td></tr><tr><td>10</td><td>PrepPlace.Advanced</td><td>A2.1</td><td>30.0</td><td>58.33%</td></tr><tr><td>11</td><td>Adjective.Comparison</td><td>A2.2</td><td>40.7</td><td>83.33%</td></tr><tr><td>12</td><td>SuffixAdjective.Basic</td><td>A2.2</td><td>42.0</td><td>62.50%</td></tr><tr><td>13</td><td>PastSimple</td><td>A2.1</td><td>32.2</td><td>55.36%</td></tr><tr><td>14</td><td>PresentParticiple</td><td>A2.2</td><td>38.0</td><td>51.79%</td></tr><tr><td>14</td><td>Adverb.Meaning</td><td>B1.1</td><td>45.0</td><td>75.00%</td></tr><tr><td>15</td><td>Prep.To</td><td>B1.1</td><td>45.0</td><td>76.79%</td></tr><tr><td>15</td><td>PastPerfect</td><td>B2.1</td><td>60.8</td><td>60.71%</td></tr><tr><td>15</td><td>PrepPlace.Basic</td><td>A1</td><td>26.0</td><td>66.67%</td></tr><tr><td>15</td><td>PassiveVoice</td><td>B1.2</td><td>56.8</td><td>62.50%</td></tr><tr><td>15</td><td>ImperativeClause</td><td>A2.2</td><td>36.3</td><td>64.29%</td></tr><tr><td>15</td><td>Conditional.1</td><td>B1.1</td><td>46.0</td><td>66.67%</td></tr><tr><td>15</td><td>SuffixAdjective.Advanced</td><td>B1.1</td><td>48.0</td><td>67.14%</td></tr><tr><td>15</td><td>VerbAgreement.Causative</td><td>B1.2</td><td>56.0</td><td>35.71%</td></tr><tr><td>15</td><td>Prep.Between</td><td>B1.1</td><td>49.0</td><td>60.71%</td></tr></tbody></table>
## B.4 Advanced

<table><caption>TABLE B.4: The proposed learning path for advanced students. NB: Several skills can be proposed at the same step of the learning path. Step 0 stands for knowledge mastered by all advanced students.</caption><thead><tr><th>Step</th><th>Attribute</th><th>CEFR</th><th>GSE</th><th>Success Rate</th></tr></thead><tbody><tr><td>0</td><td>SuffixAdjective.Basic</td><td>A2.2</td><td>42.0</td><td>81.25%</td></tr><tr><td>0</td><td>SuffixAdjective.Advanced</td><td>B1.1</td><td>48.0</td><td>95.00%</td></tr><tr><td>0</td><td>Conjunction.And</td><td>A1</td><td>27.5</td><td>93.75%</td></tr><tr><td>0</td><td>Conjunction.Cause</td><td>B1.1</td><td>50.3</td><td>93.75%</td></tr><tr><td>0</td><td>Conjunction.Contrast</td><td>B1.1</td><td>50.3</td><td>86.46%</td></tr><tr><td>0</td><td>PresentSimple</td><td>A2.1</td><td>30.8</td><td>84.38%</td></tr><tr><td>0</td><td>PrepPlace.Basic</td><td>A1</td><td>26.0</td><td>95.83%</td></tr><tr><td>0</td><td>PrepPlace.Advanced</td><td>A2.1</td><td>30.0</td><td>93.75%</td></tr><tr><td>0</td><td>Prep.To</td><td>B1.1</td><td>45.0</td><td>100.00%</td></tr><tr><td>0</td><td>PastSimple</td><td>A2.1</td><td>32.2</td><td>85.94%</td></tr><tr><td>0</td><td>Prep.Time</td><td>A2.2</td><td>37.8</td><td>99.11%</td></tr><tr><td>1</td><td>Adverb.Meaning</td><td>B1.1</td><td>45.0</td><td>96.88%</td></tr><tr><td>2</td><td>Prep.Between</td><td>B1.1</td><td>49.0</td><td>93.75%</td></tr><tr><td>2</td><td>VerbAgreement.Subjunctive</td><td>B1.2</td><td>53.0</td><td>75.00%</td></tr><tr><td>2</td><td>Conditional.23</td><td>B1.2</td><td>57.7</td><td>87.50%</td></tr><tr><td>3</td><td>PassiveVoice</td><td>B1.2</td><td>56.8</td><td>90.62%</td></tr><tr><td>4</td><td>SubjectVerbAgreement</td><td>A1</td><td>24.0</td><td>86.25%</td></tr><tr><td>4</td><td>Conditional.1</td><td>B1.1</td><td>46.0</td><td>95.83%</td></tr><tr><td>4</td><td>VerbAgreement</td><td>B1.1</td><td>50.3</td><td>78.75%</td></tr><tr><td>5</td><td>PastParticiple</td><td>A2.2</td><td>38.0</td><td>92.50%</td></tr><tr><td>5</td><td>PresentParticiple</td><td>A2.2</td><td>38.0</td><td>84.38%</td></tr><tr><td>5</td><td>PresentPerfect</td><td>B1.1</td><td>43.9</td><td>93.75%</td></tr><tr><td>5</td><td>Adjective.Comparison</td><td>A2.2</td><td>40.7</td><td>91.67%</td></tr><tr><td>6</td><td>PastPerfect</td><td>B2.1</td><td>60.8</td><td>84.38%</td></tr><tr><td>7</td><td>ImperativeClause</td><td>A2.2</td><td>36.3</td><td>87.50%</td></tr><tr><td>7</td><td>VerbAgreement.Causative</td><td>B1.2</td><td>56.0</td><td>68.75%</td></tr></tbody></table>
Bibliography

Albert, Dietrich, Martin Schrepp, and Theo Held (1994). "Construction of knowledge spaces for problem solving in chess". In: *Contributions to mathematical psychology, psychometrics, and methodology*. Springer, pp. 123–135.

Alderson, J. Charles et al. (2004). "The development of specifications for item development and classification within The Common European Framework of Reference for Languages: Learning, Teaching, Assessment: Reading and Listening: Final report of The Dutch CEF Construct Project." In: URL: https://api.semanticscholar.org/CorpusID:207929185.

Almond, Russell, Duanli Yan, and Lisa Hemat (2007). "Parameter recovery studies with a diagnostic Bayesian network model". In: vol. 35. 2. The Behaviormetric Society, pp. 159–185.

Avella, John T et al. (2016). "Learning analytics methods, benefits, and challenges in higher education: A systematic literature review." In: vol. 20. 2. ERIC, pp. 13–29.

Beck, Joseph E and Yue Gong (2013). "Wheel-spinning: Students who fail to master a skill". In: *Artificial Intelligence in Education: 16th International Conference, AIED 2013, Memphis, TN, USA, July 9-13, 2013. Proceedings 16*. Springer, pp. 431–440.

Beckmann, Jens F, Damian P Birney, and Natassia Goode (2017). "Beyond psychometrics: the difference between difficult problem solving and complex problem solving". In: vol. 8. Frontiers Media SA, p. 1739.

Bordat, Jean-Paul (1986). "Calcul pratique du treillis de Galois d'une correspondance". In: vol. 96, pp. 31–47.

Brandt, Silke, Dietrich Albert, and Cord Hockemeyer (2003). "Surmise relations between tests—mathematical considerations". In: vol. 127. 2. Elsevier, pp. 221–239.

Calò, Eduardo (2021). "Building a Learning Space for Language Acquisition". PhD thesis. IDMC (Univeristé de Lorraine).

Chapelle, Carol and Cheryl Roberts (1986). "Ambiguity tolerance and field independence as predictors of proficiency in English as a second language". In: vol. 36. 1. Wiley Online Library, pp. 27–45.

Chatti, Mohamed Amine et al. (2012). "A reference model for learning analytics". In: vol. 4. 5-6. Inderscience Publishers, pp. 318–331.

Chen, Hui et al. (2021). "Learning path recommendation for MOOC platforms based on a knowledge graph". In: *Knowledge Science, Engineering and Management: 14th International Conference, KSEM 2021, Tokyo, Japan, August 14–16, 2021, Proceedings, Part II 14*. Springer, pp. 600–611.

Chiusole, Debora de et al. (2020). "Stat-Knowlab. Assessment and learning of statistics with competence-based knowledge space theory". In: vol. 30. Springer, pp. 668–700.

Chrysafiadi, Konstantina and Maria Virvou (2013). "Student modeling approaches: A literature review for the last decade". In: vol. 40. 11. Elsevier, pp. 4715–4729.

Chung, Hyunsook and Jeongmin Kim (2016). "An ontological approach for semantic modeling of curriculum and syllabus in higher education". In: vol. 6. 5. IACSIT Press, p. 365.
Collins, Jason A, Jim E Greer, and Sherman X Huang (1996). "Adaptive assessment using granularity hierarchies and Bayesian nets". In: *International conference on intelligent tutoring systems*. Springer, pp. 569-577.

Corbett, Albert (2000). "Cognitive mastery learning in the act programming tutor". In: *Adaptive User Interfaces*. Vol. 7, p. 1.

Culbertson, Michael J (2016). "Bayesian networks in educational assessment: The state of the field". In: vol. 40. 1. SAGE Publications Sage CA: Los Angeles, CA, pp. 3-21.

Cumming, Geoff and Anne McDougall (2000). "Mainstreaming AIED into education?" In: vol. 11, pp. 197-207.

Desmarais, Michel C and Ryan SJ d Baker (2012). "A review of recent advances in learner and skill modeling in intelligent learning environments". In: vol. 22. Springer, pp. 9-38.

DiBello, Louis V and William Stout (2007). "Guest editors' introduction and overview: IRT-based cognitive diagnostic models and related methods". In: JSTOR, pp. 285-291.

Dietterich, Thomas G (1998). "Approximate statistical tests for comparing supervised classification learning algorithms". In: vol. 10. 7. MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info ..., pp. 1895-1923.

Doignon, Jean-Paul (2014). "Learning spaces, and how to build them". In: *Formal Concept Analysis: 12th International Conference, ICFCA 2014, Cluj-Napoca, Romania, June 10-13, 2014. Proceedings 12*. Springer, pp. 1-14.

Doignon, Jean-Paul and Jean-Claude Falmagne (1985). "Spaces for the assessment of knowledge". In: vol. 23. 2. Elsevier, pp. 175-196.

— (2012). *Knowledge spaces*. Springer Science & Business Media.

— (2015). "Knowledge spaces and learning spaces". In.

Dowling, Cornelia E (1993). "On the irredundant generation of knowledge spaces". In: vol. 37. 1. Elsevier, pp. 49-62.

Düntsch, Ivo and Günther Gediga (1995). "Skills and knowledge structures". In: vol. 48. 1. Wiley Online Library, pp. 9-27.

Ganter, Bernhard and Sergei O Kuznetsov (2001). "Pattern structures and their projections". In: *International conference on conceptual structures*. Springer, pp. 129-142.

Ganter, Bernhard and Gerd Stumme (2005). *Formal concept analysis: foundations and applications*. Vol. 3626. Springer Science & Business Media.

Ganter, Bernhard and Rudolf Wille (2012). *Formal concept analysis: mathematical foundations*. Springer Science & Business Media.

Ganter, Bernhard et al. (2017). "An invitation to knowledge space theory". In: *Formal Concept Analysis: 14th International Conference, ICFCA 2017, Rennes, France, June 13-16, 2017, Proceedings 14*. Springer, pp. 3-19.

Getie, Addisu Sewbihon (2020). "Factors affecting the attitudes of students towards learning English as a foreign language". In: vol. 7. 1. Taylor & Francis, p. 1738184.

Govindarajan, Kannan, Vivekanandan Suresh Kumar, et al. (2016). "Dynamic learning path prediction—A learning analytics solution". In: *2016 IEEE eighth international conference on technology for education (T4E)*. IEEE, pp. 188-193.

Guo, Lu et al. (2021). "Evolution and trends in intelligent tutoring systems research: a multidisciplinary and scientometric view". In: vol. 22. 3. Springer, pp. 441-461.

Heck, Daniel W and Stefano Noventa (2020). "Representing probabilistic models of knowledge space theory by multinomial processing tree models". In: vol. 96. Elsevier.
Heller, Jürgen et al. (2015). "On the link between cognitive diagnostic models and knowledge space theory". In: vol. 80. Springer, pp. 995-1019.

Holzknecht, Franz, Ari Huhta, and Iasonas Lamprianou (2018). "Comparing the outcomes of two different approaches to CEFR-based rating of students' writing performances across two European countries". In: vol. 37. Elsevier, pp. 57-67.

Janostik, Radek, Jan Konecny, and Petr Krajča (2022). "LCM from FCA point of view: A CbO-style algorithm with speed-up features". In: vol. 142. Elsevier, pp. 64-80.

Jourdan, Guy-Vincent, Jean-Xavier Rampon, and Claude Jard (1994). "Computing on-line the lattice of maximal antichains of posets". In: vol. 11. 3. Springer, pp. 197-210.

Kambouri, Maria et al. (1994). "Knowledge assessment: Tapping human expertise by the QUERY routine". In: vol. 40. 1. Elsevier, pp. 119-151.

Koppen, Mathieu (1993). "Extracting human expertise for constructing knowledge spaces: An algorithm". In: vol. 37. 1. Elsevier, pp. 1-20.

Koppen, Mathieu and Jean-Paul Doignon (1990). "How to build a knowledge space by querying an expert". In: vol. 34. 3. Elsevier, pp. 311-331.

Krajca, Petr, Jan Outrata, and Vilem Vychodil (2010). "Parallel algorithm for computing fixpoints of Galois connections". In: vol. 59. 2. Springer, pp. 257-272.

Kuznetsov, Sergei O (1993). "A fast algorithm for computing all intersections of objects from an arbitrary semilattice". In: 1. VSESOYUZNYI INST NAUCHNO-TEKHNICHESKOI INFORM BALTIISKAYA UL 14, 1-19 MOSCOW ..., pp. 17-20.

Lee, Hansol and Jang Ho Lee (2024). "The effects of AI-guided individualized language learning: A meta-analysis". In: University of Hawaii National Foreign Language Resource Center.

Liang, Jia-Cing et al. (2023). "Roles and research foci of artificial intelligence in language education: an integrated bibliographic analysis and systematic review approach". In: vol. 31. 7. Taylor & Francis, pp. 4270-4296.

Lin, Chien-Chang, Anna YQ Huang, and Owen HT Lu (2023). "Artificial intelligence in intelligent tutoring systems toward sustainable education: a systematic review". In: vol. 10. 1. Springer, p. 41.

Liu, Guilong (2021). "Rough set approaches in knowledge structures". In: vol. 138. Elsevier, pp. 78-88.

Liu, Peng and Zhizhong Li (2012). "Task complexity: A review and conceptualization framework". In: vol. 42. 6. Elsevier, pp. 553-568.

Loewen, Shawn, Daniel R Isbell, and Zachary Sporn (2020). "The effectiveness of app-based language instruction for developing receptive linguistic knowledge and oral communicative ability". In: vol. 53. 2. Wiley Online Library, pp. 209-233.

Lougheed, Lin (2013). "Barron's TOEIC: test of English for international communication". In.

Lyster, Roy (1998). "Recasts, repetition, and ambiguity in L2 classroom discourse". In: JSTOR, pp. 51-81.

Matthiessen, Christian (May 2015). "Reflections on "Researching and Teaching Chinese as a Foreign Language"". In: vol. 1, pp. 1-27. DOI: 10.1558/rtcfl.vii1.27271.

Mousavinasab, Elham et al. (2021). "Intelligent tutoring systems: a systematic review of characteristics, applications, and evaluation methods". In: vol. 29. 1. Taylor & Francis, pp. 142-163.
Muangprathub, Jirapond, Veera Boonjing, and Kosin Chamnongthai (2020). "Learning recommendation with formal concept analysis for intelligent tutoring system". In: vol. 6. 10. Elsevier.

Müller, Cornelia E (1989). "A procedure for facilitating an expert's judgements on a set of rules". In: *Mathematical psychology in progress*. Springer, pp. 157-170.

Nabizadeh, Amir Hossein et al. (2020). "Learning path personalization and recommendation methods: A survey of the state-of-the-art". In: vol. 159. Elsevier.

Nguyen, Ha and Yongqi Gu (2020). "Impact of TOEIC listening and reading as a university exit test in Vietnam". In: vol. 17. 2. Taylor & Francis, pp. 147-167.

Nicholson, Simon James (2015). "Evaluating the TOEIC® in South Korea: Practicality, reliability and validity". In: vol. 7. 1. Citeseer, pp. 221-233.

Nkambou, Roger, Riichiro Mizoguchi, and Jacqueline Bourdeau (2010). *Advances in intelligent tutoring systems*. Vol. 308. Springer Science & Business Media.

Noh, Norzaidah Md et al. (2012). "Intelligent tutoring system using rule-based and case-based: a comparison". In: vol. 67. Elsevier, pp. 454-463.

North, Brian (2005). "The CEFR levels and descriptor scales". In: *Multilingualism and assessment: Achieving transparency, assuring quality, sustaining diversity*. Proceedings of the ALTE Berlin Conference, pp. 21-66.

Nourine, Lhouari and Olivier Raynaud (1999). "A fast algorithm for building lattices". In: vol. 71. 5-6. Elsevier, pp. 199-204.

Nwana, Hyacinth S (1990). "Intelligent tutoring systems: an overview". In: vol. 4. 4. Springer, pp. 251-277.

Outrata, Jan and Vilem Vychodil (2012). "Fast algorithm for computing fixpoints of Galois connections induced by object-attribute relational data". In: vol. 185. 1. Elsevier, pp. 114-127.

Pelánek, Radek (2017). "Bayesian knowledge tracing, logistic models, and beyond: an overview of learner modeling techniques". In: vol. 27. Springer, pp. 313-350.

— (2020). "Managing items and knowledge components: domain modeling in practice". In: vol. 68. 1. Springer, pp. 529-550.

Pelánek, Radek, Tomáš Effenberger, and Jaroslav Čechák (2022). "Complexity and difficulty of items in learning systems". In: vol. 32. 1. Springer, pp. 196-232.

Rautionaho, Paula, Sandra C. Deshors, and Lea Meriläinen (2018). "Revisiting the ENL-ESL-EFL continuum: A multifactorial approach to grammatical aspect in spoken Englishes". In: vol. 42. 1, pp. 41-78. DOI: doi:10.1515/icame-2018-0004. URL: https://doi.org/10.1515/icame-2018-0004.

Samejima, Fumiko (2012). "Some methods and approaches of estimating the operating characteristics of discrete item responses". In: *Principals of modern psychological measurement*. Routledge, pp. 159-182.

Sarmah, Achyanta Kumar, Shyamanta M Hazarika, and Smriti Kumar Sinha (2015). "Formal concept analysis: current trends and directions". In: vol. 44. Springer, pp. 47-86.

Schrepp, Martin and Theo Held (1995). "A simulation study concerning the effect of errors on the establishment of knowledge spaces by querying experts". In: vol. 39. 4. Elsevier, pp. 376-382.

Self, John (1988). "Bypassing the intractable problem of student modelling". In: pp. 18-24.

Shalini and Ankit Tewari (2020). "Sustainable education in India through artificial intelligence: Challenges and opportunities". In: *Companion Publication of the 12th ACM Conference on Web Science*, pp. 41-47.
Shao, Zengzhen et al. (2020). "Research on a new automatic generation algorithm of concept map based on text analysis and association rules mining". In: vol. 11. 2. Springer, pp. 539-551.

Shemshack, Atikah, Kinshuk, and Jonathan Michael Spector (2021). "A comprehensive analysis of personalized learning components". In: vol. 8. 4. Springer, pp. 485-503.

Shi, Daqian et al. (2020). "A learning path recommendation model based on a multi-dimensional knowledge graph framework for e-learning". In: vol. 195. Elsevier, p. 105618.

Silva, Felipe Leite da et al. (2023). "A systematic literature review on educational recommender systems for teaching and learning: research trends, limitations and opportunities". In: vol. 28. 3. Springer, pp. 3289-3328.

Spearman, Charles (1961). "The proof and measurement of association between two things." In: Appleton-Century-Crofts.

Spoto, Andrea, Luca Stefanutti, and Giulio Vidotto (2010). "Knowledge space theory, formal concept analysis, and computerized psychological assessment". In: vol. 42. 1. Springer, pp. 342-350.

— (2016). "An iterative procedure for extracting skill maps from data". In: vol. 48. Springer, pp. 729-741.

Stankov, Slavomir, Vlado Glavinić, and Marko Rosić (2000). "On knowledge representation in an intelligent tutoring system". In: *Proc. 4th IEEE International Conference on Intelligent Engineering Systems*, pp. 17-19.

Stefanutti, Luca et al. (2020). "Modeling misconceptions in knowledge space theory". In: vol. 99. Elsevier, p. 102435.

Stumme, Gerd et al. (2002). "Computing iceberg concept lattices with titanic". In: vol. 42. 2. Elsevier, pp. 189-222.

Sun, Wen et al. (2021). "Knowledge structures delineated by fuzzy skill maps". In: vol. 407. Elsevier, pp. 50-66.

Talcott, C. and G. Tullis (2007). *Target Score Teacher's Book: A Communicative Course for TOEIC® Test Preparation*. Cambridge professional English v. 3. Cambridge University Press. ISBN: 9780521706650. URL: https://books.google.fr/books?id=oinGxQEACAAJ.

Uno, Takeaki, Masashi Kiyomi, Hiroki Arimura, et al. (2004). "LCM ver. 2: Efficient mining algorithms for frequent/closed/maximal itemsets". In: *Fimi*. Vol. 126.

Uno, Takeaki et al. (2003). "LCM: An Efficient Algorithm for Enumerating Frequent Closed Item Sets." In: *Fimi*. Vol. 90.

Woolf, Beverly Park (2010). *Building intelligent interactive tutors: Student-centered strategies for revolutionizing e-learning*. Morgan Kaufmann.

Zaki, Mohammed J and Ching-Jui Hsiao (2002). "CHARM: An efficient algorithm for closed itemset mining". In: *Proceedings of the 2002 SIAM international conference on data mining*. SIAM, pp. 457-473.