problems have shown that subjects transform them into well-defined problems in order to reach resolution. This process consists first of all in specifying the starting state and the goal state by elaborating precise sub-problems, and it is the resolution of each of these sub-problems that would lead to the resolution of the problem. Moreover, this type of problem solicits both general and specific knowledge. Thus, a learner who has specific knowledge about the information in the problem has less difficulty in establishing the sub-problems. As for general knowledge, it is mobilized when the learner does not have specific knowledge about certain information of the problem. The CreaCube task is of this type.

"A well-defined problem is one in which the starting situation [(initial state)] and the goal to be achieved [(goal state)] are clearly stated" (Lemaire and Didierjean, 2018). In a well-defined problem, the criteria derived from the goal to be reached allow the solution to be evaluated. It is the achievement of the goal that allows us to say that the problem is solved. By studying the resolution of this type of problem, researchers wish to identify the sequence of mental operations involved in the resolution process. Patrick Lemaire cites the "means-ends" analysis strategy developed by Newell and Simonen 1972 in the General Problem Solver (GPS) theory. "The means-ends analysis consists in comparing the present state of the problem and its target state and then selecting an operator that will reduce the difference" (Lemaire and Didierjean, 2018). It is a recursive process that involves, on one hand, the analysis of the difference and on the other, the selection of operators allowing the reduction of the difference between the starting state and the goal state until the solution of the problem is obtained. This strategy assumes two types of skill:

* "the skill to detect the difference between the desired state and the current state of the problem" (Lemaire and Didierjean, 2018);

* "the skill to implement an action that will reduce this difference" (Lemaire and Didierjean, 2018).

<figure><img src="image_18.png" /><figcaption>**Figure 17 - Stages of problem solving according to the cognitivist theory.**
This figure represents the stages of problem solving according to the cognitivist theory and more precisely the General Problem Solver theory.</figcaption></figure>

The General Problem Solver theory is a computational theory (it has been implemented and tested by simulations). It has been designed in such a way as to allow the resolution of various problems based on very precise postulates about cognitive processes. It implements four stages of information processing that lead to the resolution of the problem posed<sup>81</sup>. These stages are represented in figure 17 and explained below:

1. **The representation of the problem.** It is during this step that the problem space is 81This structuring has influenced different cognitive architectures discussed by Antonio Lieto (Lieto, 2021; Lieto et al., 2018) but will not be discussed here.
constructed. The problem space is composed of the initial state, the final state of the problem as well as its constraints. It allows the retrieval of pertinent information in long-term memory (identification of the similarity between this situation and a previously experienced situation)⁸². This space is constructed by the learner and not given to him/her at the outset with the subject of the problem. It will be affected by everything the learner knows about the problem and its resolution.

2. **The selection of the operator.** During this step, the learner looks for a pertinent operation to perform in order to change the initial state of the problem. It is possible that sets of operators are already associated in the problem space and some can be selected or invented. The smaller the problem space, the easier it is to identify the appropriate operators. Conversely, the larger the problem space, the more difficult it is to find the operator and the more the learners prefer to use heuristics⁸³. The choice of operator is conditioned by the level of reduction of the gap between the initial state and the goal state (the more the operator reduces the difference between the two states, the more adequate it is considered to be).

3. **The application of the selected operator.** A new state of the problem is reached here which may or may not correspond to the goal state, and which may or may not be close to it. The goal state can be reached with only one operator or can require several.

4. **The evaluation of the problem state.** The state reached, by the action of the previously selected operator, is evaluated here. The current state can correspond to the goal state and the learner considers that the problem is solved. Conversely, the current state differs from the goal state and the learner resumes the resolution process at step 1 (he re-evaluates his representation of the problem) or at step 2 (he searches for and selects a new operator).

"The implementation of each of these steps is strongly influenced by general constraints on human cognition, such as the limits of working memory or as declarative and procedural knowledge stored in long-term memory, as well as by certain individual characteristics" [Lemaire and Didierjean, 2018] of learners.

The General Problem Solver model consists of two essential components in the problem solving process: **the representation of the problem** and **the search for the solution**.

The first component, **the problem representation**, has been modeled in the context of the *General Problem Solver* theory. The resulting modeling is called UNDERSTAND. "*It is a reading program that extracts the deep structure (syntactic and semantic) of sentences and builds, from there, a global description of the problem and its parts*" (Lemaire and Didierjean, 2018).

The second component, **the search for the solution**, is divided into two methods:

<footnote>82We are talking here about analogy: "analogy is a heuristic that looks for similarities between a problem to be solved and a problem solved in the past" (Lemaire and Didierjean, 2018).</footnote>

<footnote>83"A heuristic is a rule of unsystematic actions or a general strategy that can lead to an answer (correct or not) quite quickly" (Lemaire and Didierjean, 2018).</footnote>
algorithms<sup>84 85</sup> and heuristics.

Backward search, analogy, and means-ends analysis are examples of heuristics. The last two (analogy and means-ends analysis) have already been discussed earlier. The first example, backward search, is limited to a certain class of problems with a clearly defined goal. This form of heuristic "starts by identifying the goal to be reached, the initial state of the problem, and seeks to go from the goal to the initial state" (Lemaire and Didierjean, 2018) (for example, when the problem to be solved is a maze). This type of search can also be used to better identify easy-to-reach sub-goals.

Problem-solving requires creativity, especially when looking for operators. However, this requires the ability to explore new paths by breaking out of the classical patterns of thinking. Creativity and thus problem-solving can be hindered by two obstacles: **functional fixedness** and **contextual anchoring** (also called the Einstellung effect).

Functional fixedness is the inability to assign other functions to an object than its usual function. As in the candle problem<sup>86</sup>, it is to be able to discover other possible functions of an object in a given context, and therefore other affordances in the sense defined in this document. When an individual can overcome this limit, they can be more creative and solve more complicated problems.

Contextual anchoring or the Einstellung effect is a "mental fixation effect observed in problem solving. To solve a problem, subjects use the same strategy as the one used for a series of previous problems, even if the strategy is not the most adapted to solve this particular problem" (Lemaire and Didierjean, 2018). The learner then finds it difficult to detach himself from the solution he already knows, whereas another, more creative solution would be more pertinent and effective. This first idea is fixed, anchored in the context.

## From the present state to the goal state

Throughout this work, there has been talk of goal, sub-goal, and goal state. However, the term goal has not been precisely defined, even if common sense allows us to understand the elements discussed above. Here, it seems necessary to give a definition. "A goal is everything that an individual strives to achieve. [...] More specifically, a goal is a future-focused cognitive representation of a desired end state that guides behavior" (Reeve, 2017).

As stated earlier, problem solving involves reducing the difference between the present state and the target state (the goal). This difference corresponds to a gap, a divergence, an incongruity. This would have motivational properties that make it possible to remedy this incongruity by developing an action plan in order to reach the goal state and thus regain congruity.

The Test-Operate-Test-Exit model represents the cognitive mechanism "by which plans drive and direct behavior" (Reeve, 2017). The first step (test) of this model corresponds to an evaluation of the current state versus the goal state. Incongruence occurs

<footnote>84"An algorithm is a rule or sequence of actions that, if applied correctly, necessarily results in a correct answer" (Lemaire and Didierjean, 2018).</footnote>

<footnote>85"The search in the problem space can implement two algorithms: a random search or a systematic search" (Lemaire and Didierjean, 2018).</footnote>

<footnote>86In Karl Duncker's candle problem https://en.wikipedia.org/wiki/Candle Problem, an object must be used in an unconventional way to solve a seemingly insoluble problem.</footnote>
if there is a discrepancy between the two states, which leads to the development of an action plan (the "operate" stage. It is when the actions implemented allow to obtain the state of congruity at the next test that the process stops (step "leave"). Conversely, when the first test phase does not reveal any discrepancies, there is congruence, which allows the process to go directly to the "exit" stage without having to perform any actions.

According to Johnmarshall Reeve, there are two types of gaps: **gap reduction** and **gap creation**.

The first type is based "on feedback that detects divergence and underlies action plans and corrective motivation" (Reeve, 2017). This feedback is in fact feedback from the environment regarding the learner's level of performance in terms of the difference between the current state and the goal state. It is a negative feedback loop that reduces discrepancies: "a discrepancy occurs, an action is taken, and the negative feedback (the discrepancy is reduced) ends that action" (Reeve, 2017).

The second type is based on a proaction system "in which the person envisions and defines in advance a higher-level future goal" (Reeve, 2017) that does not yet exist in the problem space. In this case, it is a positive feedback loop: "a discrepancy is created, an action is taken, and the positive feedback energizes further discrepancy creation" (Reeve, 2017). This loop expands the divergence.

Divergence or incongruence (or gap) is the basis of the motivation that generates action, whatever its type. Gap reduction corresponds to **corrective motivation**, it is reactive, seeks to overcome the deficiency and refers to a feedback system. On the other hand, gap creation corresponds to **goal-setting motivation**, it is proactive, seeks growth and refers to a prevention system.

"Corrective motivation activates a decision-making process, in which the individual considers many ways to reduce the present-ideal mismatch: change the plan, change the behavior (increase the effort), or leave the plan altogether" (Reeve, 2017). It is a dynamic and flexible process that pushes the learner to pursue the most appropriate action. Corrective motivation is what allows the action plan to be re-evaluated in order to achieve problem resolution.

Goal setting "generates motivation by focusing [learners'] attention on the discrepancy (or incongruence) between their present level of achievement [...] and their ideal level of achievement [...]. Researchers refer to this divergence [...] as the 'goal-performance gap'
<figure><img src="image_19.png" /><figcaption>Figure 18 - Problem solving based on the General Problem Solver theory and the Test-Operate-Test-Exit model.</figcaption></figure>

This figure takes again figure 17 based on the General Problem Solver theory by adding the elements of the TOTE (Test-Operate-Test-Exit) or TOTQ in French (Tester-Operate-Tester-Quitter) model.

Taking into account all these elements, it is possible to establish a parallel between the General Problem Solver theory and the Test-Operate-Test-Exit model. For this purpose, figure 17 has been reworked to try to integrate this new information. Figure 18 is the result. The steps of the problem solving have been preserved but some processes have been clarified. Indeed, at the end of the evaluation, the incongruity (discrepancy between the present state and the goal state) or the congruity (concordance between the two states) are put forward and allow to reach a conclusion on the state of the problem: solved or not solved, goal reached or not reached. The most difficult part is to identify when the negative and positive feedback intervene. The "operate" stage of the model could correspond to stage 2 of the theory behind this figure. From the "problem not solved" state to the return to step 2, it would be the negative feedback process that would intervene. Indeed, it refers to the corrective motivation that allows the action plan to be re-evaluated and therefore new operators to be selected. On the other hand, in a less obvious way, it would be the positive feedback that would lead the learner to return to step 1 at the end of the evaluation. This refers to goal-setting motivation. Indeed, it is during the representation of the problem that goals are identified. However, in this type of feedback, each new goal set must be more difficult than the previous one. If, at the end of the evaluation, the problem is not solved but the goal state is not different from the previous one, it is not possible to speak of positive feedback. Moreover, if returning to Step 1 leads to setting a new goal, it will necessarily be a lower level⁸⁷ to make it easier to resolve the problem. Therefore, the positive feedback could be true for setting new sub-goals leading to the goal state, provided that this end state has not been set up front⁸⁸. In view of this, the model depicted in Figure 18 is not entirely correct.

Regarding the previously mentioned reservations about the positive feedback process, it was necessary to revise Figure 18. It is in figure 19 that this has been done. It is possible to see the notion of sub-goal appearing there. In order to reach the goal state (which corresponds to the resolution of the problem), it will be necessary to identify, during the representation of the problem, a sub-goal that is easier to reach than the goal state. The evaluation of the implementation of the plan establishes whether there is incongruity or congruity. In the first case, the sub-goal is not achieved. There is then a negative feedback leading to the revision of the plan through corrective motivation. In the second case, congruence refers to the achievement of the subgoal. This achievement can lead to two new states through two new processes. If the subgoal is reached and there is congruence with

87It will be an intermediate step leading to the goal state.

88That is, goals should be set so that their difficulty increases with each positive feedback.
the goal state, then the problem is solved. Conversely, it is the positive feedback process that comes into play leading to a new representation of the problem with the identification of a new and more difficult sub-goal. This process involves goal-setting motivation.

<figure><img src="image_20.png" /><figcaption>Figure 19 - Problem solving based on the General Problem Solver theory and the Test-Operate-Test-Exit model: improved version.</figcaption></figure>

This figure takes again figure 18 based on the General Problem Solver theory with the elements of the TOTE (Test-Operate-Test-Exit) or TOTQ model in French (Tester-Opérer-Tester-Quitter) and by integrating the distinction between goal and sub-goal.

**Problem-solving: conclusion**

This part allowed us to establish the theoretical framework of problem solving by addressing the Gestalt and cognitivist currents. It emerged that the Gestalt approach was a relative precursor to other research conducted in this field. However, it is the cognitivist approach that is the most pertinent to the subject under study. This approach allowed us to identify and define the notions essential to the understanding of the problem solving process as well as the importance of goals in this process.

In the context of the CreaCube problem-solving task, the problem space is constructed by the learner in order to represent the elements and means to be used to reach the goal state: to assemble the four cubes to constitute a vehicle that moves autonomously from point A to point B. It establishes the sub-goals to be reached to constitute this vehicle. For example, a vehicle has wheels so the sub-goal is to find wheels. The plan of action is then to explore, manipulate the cubes to look for and identify wheels. It can reach this sub-goal and thus arrive by positive feedback at the setting of a new sub-goal after having proceeded to a new representation of the problem: for example, a vehicle has an engine, the sub-goal then consists in finding an engine. Conversely, the learner may not reach this sub-goal, calling on corrective motivation which will allow him/her to review and refine his/her plan of action: for example, taking each cube one by one, exploring each face... Once this process has been repeated several times, the learner may reach the last sub-goal which will then correspond to the goal state and lead to the success of the task.
Within this work, and even more so in this part, the notion of motivation appeared as transversal and essential to solve a problem-solving task. Here, it was a question of corrective motivation and goal-setting motivation. The notions of intrinsic and extrinsic motivation, curiosity and commitment were also mentioned in the previous sections. They are at the heart of the subject treated here and more specifically of the subject treated by the FLOWERS team. They will therefore be discussed in more detail, in particular through the analysis of the work carried out within the FLOWERS team.

## 2.2 Motivation

### 2.2.1 General Information

Alain Lieury explains, in 2020, that different terms in everyday language are used, more or less indifferently, to talk about motivation without having the same meaning. He therefore proposes to define motivation "as the set of biological and psychological mechanisms that allow the triggering of action, orientation (towards a goal or, conversely, away from it) and finally intensity and persistence: the more motivated one is, the greater and more persistent the activity" (Lieury & Léger, 2020). This makes the link with the different elements discussed in the previous sections on cognitive processes and the mechanisms of problem-solving.

The author had already used this definition in another work in 2017 (Lieury, 2017) and in which he differentiated two types of needs: innate needs or primary needs in connection with biological mechanisms (hunger, thirst...) and learned needs or motivations qualified as secondary rather in connection with psychological mechanisms. This second type of needs would result from learning or at least would become more complex with learning.

Jean-Philippe Abgrall defines motivation as a dynamic process resulting from two needs: **perceived competence** and **self-determination**. Perceived competence would be "the representation⁸⁹ that one has of one's cognitive abilities, i.e. metacognition" (Abgrall, 2012). Self-determination would correspond to the "representation of oneself in society, in one's environment" (Abgrall, 2012).

He specifies that emotions play a decisive role in these two types of representation. "Positive" emotions will favor positive representations and a fortiori motivation. Conversely, "negative" emotions will be the source of avoidance strategies (in opposition to motivation)⁹⁰. In this case, "perceptions are generally distorted by the overly important affective side which takes precedence over the reality of the cognitive and the negation of the person" (Abgrall, 2012). The representations that the learner has of himself within his environment as well as those that he attributes to others⁹¹ have an important influence on the learner's cognitive

90Emotions, their classification, their level of intensity as well as their mechanisms were discussed in section 1.3.7. In particular, their impact on cognitive processes, decision making and learning in which motivation is involved is discussed.

91What he thinks others think of him.
engagement in the resolution of a learning task.

This commitment will also be a function of "his orientations, his general choices, his convictions" (Abgrall, 2012) and determines a positive or negative personal involvement: this is called the conative.

To summarize, according to Jean-Philippe Abgrall, motivation is influenced by the conative, the cognitive and the affective functions.

Furthermore, Fabien Fenouillet has studied in detail different theories of motivation in order to establish precise definitions. He begins by giving a simplified definition based on behavioral impact: "motivation is what explains the dynamism of behavior" (Fenouillet, 2016). With this definition, he returns to the etymological meaning of the word motivation⁹², according to which motivation would explain, justify, support and make a behavior, an action, a movement persist.

By confronting several theories and definitions with this simplified definition, the author arrives at a general definition that takes into account the individuality of motivation: "motivation designates a hypothetical protean intra-individual force⁹³, which can have multiple internal and/or external determinants, and which makes it possible to explain the direction, the triggering, the persistence and the intensity of the behavior or action" (Fenouillet, 2016).

The author refers to motivation as an "internal force" with various effects belonging to one of four categories:

* **Direction.** "Motivation is a force that directs the individual towards certain goals. Motivated behavior has meaning that can be interpreted or analyzed in terms of the outcome(s) produced" (Fenouillet, 2016).

* **Triggering.** "One of the most visible effects of motivation is related to behavioral change. [...] Triggering is therefore not related to a simple behavioral adjustment, but reveals the presence of a new motivation" (Fenouillet, 2016).

* **Persistence.** "Adopting a behavior over time can be explained from a motivational point of view from the moment that its maintenance requires the voluntary exercise of a certain force⁹⁴ ⁹⁵. [...] Explaining persistence therefore consists in understanding the nature of this will to make the action or behavior last" (Fenouillet, 2016).

* **Intensity:** "Intensity is definitely the least ambiguous motivational effect. The production of an effort is necessarily explained by the presence of a force. [...] An individual can produce an effort with the objective of appearing motivated .... In this case, the aim is to appear motivated, regardless of the purpose of the activity considered" (Fenouillet, 2016).

<footnote>92From the Latin *moveo* meaning to move</footnote>

<footnote>93Which can take many forms.</footnote>

<footnote>94"Volition theories rely on the concept of will⁹⁵ which is a form of motivation that explains why the individual maintains his action over time to achieve one or more goals</footnote>

<footnote>95To explain the persistence of action; self-determination theory believes that the self seeks to satisfy its basic needs" (Fenouillet, 2016).</footnote>
Fabien Fenouillet explains that even if the effects produced, mentioned above, allow to partially characterize motivation, they do not allow to define its nature. "This is a problem made all the more complex by the fact that many motivational theories rely on conceptualizations in terms of purpose, interest, desire, need, or curiosity to talk about motivation" (Fenouillet, 2016). It seems then difficult to establish a global definition of motivation that would make it possible to explain its nature without being specific to a particular concept. The author then proposes to give a theoretical definition that would be based on the most general categories possible. In particular, he cites the example of the theory of self-determination, which categorizes motivation as being either intrinsic or extrinsic. "Motivation assumes not only that there is a motive for the individual's behavior but also that the individual is able to anticipate the effects of that behavior if he or she decides to act" (Fenouillet, 2016). This definition is part of an integrative model (represented in figure 20) aiming to "propose a global classification of motivational theories and is based on relatively general categories" (Fenouillet, 2016).

The author defined seven conceptual sets grouping the conceptual categories of
motivation:

1. **Primary motive.** "Primary motives, which aim to explain the origin of motivation from a psychological point of view, are of two kinds: **instincts** and **needs**" (Fenouillet, 2016). The instinct is what pushes an individual to make an action directed towards a goal of which he is not conscious. The needs can be physiological (hunger, thirst, etc.) or psychological (in connection with the social relations for example).

a.

2. **Secondary motive.** They are to be differentiated from primary motives because "they do not try to explain the absolute origin (from the psychological point of view) of the behavior and they admit that the environment can contain factors able to be at the origin of the motivation" (Fenouillet, 2016). This set gathers twelve conceptual categories⁹⁶: **value**, **goal**, **interest**, **self-esteem**, **drive** (energy, dynamism), **dissonance**, **emotion**, **curiosity**, **intention**, **personality trait** and **original motives**.

a.

3. **Prediction.** Prediction is one of the two facets of the goal⁹⁷. The goal referring to prediction is in fact the anticipation of the realization of a final state. The subjective probability of success or failure is based on this anticipation: the term expectation is used to describe this. This set comprises four conceptual categories: **expectation-value relationship**⁹⁸, **expectation-value distinction**⁹⁹, **control expectation**¹⁰⁰ and **imaginary prediction**¹⁰¹.

96These twelve categories will not be defined here. Some of these concepts have been discussed and defined in the previous sections and others will be discussed in more detail later in this work (such as goal-directed motivation or curiosity).

97The goal can also refer to the anticipation of the final state. It is then a goal in the sense of a motive.

98 "Probability that a given level of effort will produce a certain level of performance. [...] The term instrumentality [qualifies] a second-level expectation that establishes the probability that a level of performance will produce a given outcome" (Fenouillet, 2016).

99Here, expectation is used as a synonym for "self-efficacy". It is defined as "an individual's belief in his or her ability to organize and execute the course of action required to produce desired outcomes" (Fenouillet, 2016). The impact on value is no longer considered in the relationship between expectation and the probability of an outcome occurring.

100"Outcome expectation is subject to beliefs of control [...] This belief of control specifies the extent to which the self can produce desired events or prevent undesired events" (Fenouillet, 2016).

101 It is here a subjective component of the prediction that is considered. It is a prediction that is not
4. **Decision-making.** In this set, it is "*the selection of an action by the individual among all possible alternatives*" (Fenouillet, 2016) that is presented as one of the issues of motivation. Volition is here an explanatory variable of decision making and is associated with motivation to make a choice. This set includes two conceptual categories: **commitment**¹⁰² and **decision**¹⁰³.

5. **Strategy.** This set goes beyond the theoretical aspects treated in the previous sets. It emphasizes that setting a goal does not allow motivation to influence performance if the way to achieve it is not identified. "*If the individual does not know the best strategy to implement, motivating him or her through the assignment of a goal may even be counterproductive*" (Fenouillet, 2016). This set integrates two conceptual categories: **the cognitive strategy**¹⁰⁴ and **the emotional strategy**¹⁰⁵.

6. **Behavior.** "*The term behavior has a dual meaning: in a narrow sense, it is primarily referred to the action that the subject performs on a situation (the reaction to a stimulus in the behaviorist model), in a broader sense, the term behavior refers to all psychological functions and processes*" (Fenouillet, 2016). To understand motivation, it would be necessary to understand the intention behind the observable behavior¹⁰⁶. Motivation triggers, directs, increases the intensity and persistence of behavior.

7. **Result.** The result corresponds to "*the outcome of an action or behavior*" (Fenouillet, 2016). When evaluating a result, motivation can be one factor among others (factor of performance, resignation or well-being). Other results are "*explicable only in relation to a motivational explanation*" (Fenouillet, 2016). Flow (or flux) is a form of result. It is "*an optimal experience arising when skills are neither exceeded nor underused.* [...] *Flow is conceived as an experience that provides an intrinsic reward*" (Fenouillet, 2016).

These sets are represented by the purple squares in Figure 20. The relationships between each of these sets are shown by the different colored arrows.

**Self-regulation** (in yellow) acts at three levels of motivation: the secondary motive (that of action), decision making (the means, costs and benefits of the choice made are based on objective probabilities but on beliefs derived from past experiences.

102It is necessary to differentiate the pre-decisional phase from the post-decisional phases. It is during the first that "*the individual deliberates on the different motivations that push him to act*" (Fenouillet, 2016). During the post-decisional phases "*the individual's reflection is no longer to know why he acts but how to do it. In other words, once the individual has taken the decision to engage in action, he no longer questions the motivation that led him to the action. He is committed*" (Fenouillet, 2016). The notion of persistence and prolongation in the activity have a central place in the concept of commitment.

103"*The principle of decision [...] seems to be based on an evaluation of costs and benefits that can take multiple forms [...]. This evaluation can be understood as a calculation, which supposes that Man is a rational being*" (Fenouillet, 2016). This is based in particular on human reasoning.

104"*Cognitive strategies are plans, generally explicit means, that an individual implements to achieve an end*" (Fenouillet, 2016). They are implemented when effort alone is not enough to achieve a performance.

105Emotional strategies "*are plans or means, more or less explicit to the individual, implemented to manage emotions in general and anxiety in particular*" (Fenouillet, 2016).

106"*Measuring student motivation solely on academic performance, for example, may suggest that the student is not motivated if performance is not increasing when it may only be the effect of behavior on performance that is not visible*" (Fenouillet, 2016).
reconsidered) and at the level of strategies (modification of strategies to ensure optimal learning or effective stress management).

**Satisfaction** (in green) is a return of the result to the initial motive: the individual is satisfied when the result responds to the motive initially identified.

**Orientation** (in blue) is the opposite relationship of satisfaction: it starts from the motives and goes towards the result. This relation explains the orientation of the behavior (the motive would then orient the behavior).

**Expectation** (in gray) goes from the prediction to the result. It corresponds to the latency between the elaboration of the probabilities of failure or success and their verification. **The evaluation** (in orange) is the feedback allowing to adapt these predictions or to adjust the behavior. These two relationships have a strong impact on motivation.

<figure><img src="image_21.png" /><figcaption>Figure 20 - Integrative model of motivation.</figcaption></figure>

This figure is inspired by a diagram made by Fabien Fenouillet (Fenouillet, 2016) and represents the integrative model of motivation.

Figure 20 is divided into two parts referring to **motivation** and **volition**. These two parts are in fact two stages of motivation.

The motives and the prediction are part of the purely motivational stage. Decision-making is a boundary that separates it from the volitional stage. It is the orientation of the behavior (goal, need, interest, etc.) that comes into play here. This allows us to understand what is at the origin (trigger) of this behavior as well as its intensity.

The second step has its source at the decision-making level. "*The central question, for many volitional models, is to understand what this will is made of which makes the behavior persist in order to achieve the result which effectively motivates the individual*" (Fenouillet, 2016).

These two notions can be linked to that of desire and intention in Michael Bratman's
belief-desire-intention model¹⁰⁷ to reflect the balance between spending time deliberating in order to choose a course of action and investing in carrying it out (Rao, 1995).

On the other hand, like Fabien Fenouillet, Alain Lieury differentiates intrinsic motivation from extrinsic motivation. The latter could be related to reinforcement, that is, an individual is more motivated to complete a task the greater the reward. Conversely, "*the intrinsic motivations (curiosity, manipulation) would only have an interest in the activity itself*" (Lieury and Léger, 2020).

### 2.2.2 Extrinsic motivation

According to Johnmarshall Reeve, extrinsic motivation "results from environmental incentives and consequences [internal and/or external.] [It] comes from a consequence that is separate from the activity itself" (Reeve, 2017).

Johnmarshall Reeve differentiates three types of needs involved in extrinsic motivation:

* **Physiological needs.** A physiological need is a "biological condition within the body that synchronizes its brain structures, hormones, and major organs to regulate bodily well-being and correct bodily imbalances that may be threats to growth, development, and life" (Reeve, 2017). Hunger and thirst are physiological needs. Extrinsic motivation then results from in-citations in the person's internal environment and having consequences on that same environment.

* **Psychological needs.** A psychological need is "an innate psychological process that underlies the proactive desire to seek interactions with the environment that promote personal development, social development, and psychological well-being" (Reeve, 2017). Autonomy is an example of a psychological need.

* **Implicit motives.** An implicit motive is "a developmentally acquired process (socialization) that drives one to seek out and spend time experiencing events that have been associated with positive emotions during the person's socialization history" (Reeve, 2017). Success is an example of an implicit motive.

In contrast to physiological needs, psychological needs and implicit motives require interactions with the external environment and involve the needs for competence and social belonging. It is psychological needs and implicit motives that are particularly involved in learning tasks such as problem-solving.

The author cites Baldwin and Baldwin's 1986 model of operant conditioning. Operant conditioning is a process by which the learner learns to function effectively in the environment. That is, "*learn and engage in behaviors that produce attractive consequences [...] and also behaviors that avoid unpleasant consequences*" (Reeve, 2017).
$$S : R \rightarrow C \qquad (1)$$

Equation 1 is the conceptualization of behavior according to the operant conditioning model where:

* S is the situational signal (or stimulus).

* R is the behavioral response.

* C is the consequence.

* The colon between S and R "shows that the situational signal sets the opportunity for the behavioral response (but does not cause it)" (Reeve, 2017).

* The arrow between R and C shows that it is the behavioral response that causes the consequence.

* S : R "explains the motivational scope of incentives. [...] Incentives solicit, encourage, and bribe people to engage in behaviors that they otherwise would not have undertaken" (Reeve, 2017).

* R → C shows the motivational scope of the consequences.

This model shows that extrinsic motivation is lead by a stimulus (stimulus driven) perceived in the environment (internal and/or external).

On the other hand, Alain Lieury uses the theory of self-determination of Edward Deci et Richard Ryan to differentiate four types of extrinsic motivation. This theory is not only based on the principle that humans need self-determination and that this is involved in all forms of motivation (intrinsic and extrinsic) but also in amotivation, which is equivalent to the absence of autonomy or to constraint. Moreover, self-determination would be modulated by the need for perceived competence as well as by the need for social belonging (notions discussed in section 2.2.1). All this is summarized in Figure 21.

<figure><img src="image_22.png" /><figcaption>Figure 21 - Edward Deci and Richard Ryan's theory of self-determination.</figcaption></figure>

This figure illustrates the theory of self-determination. It shows that "the different forms of motivation are determined by a continuum of self-determination" (Lieury, 2017). The latter is modulated by the need for competence and the need for social belonging.

In this figure, the feeling of
autonomy (or self-determination) is represented by a large blue arrow and starts from the lowest level which corresponds to the constraint to reach the highest level, autonomy. As said previously, amotivation represents the lowest level, that is to say demotivation: "when the individual no longer sees any relationship between what he does and the results of his activity or that this activity makes no sense to him" (Lieury, 2017). The highest level of autonomy corresponds to intrinsic motivation (which will be discussed in more detail in section 2.2.3). Extrinsic motivation is at an intermediate level of self-determination. The individual is then regulated in different ways by the external environment.

Extrinsic motivation would then be distinguished in four forms:

* **External regulation:** "the individual is regulated by external reinforcements (the student goes to school by legal obligation)" (Lieury, 2017).

* **Introjected regulation:** the rules at the origin of this regulation are no longer external but internal (the learner does an action for fear of disappointing for example).

* **Identified regulation:** the activity carried out by the learner has value for him and he knows it, he has identified it.

* **Integrated regulation:** the activity is not interesting but corresponds to the learner's deepest values (e.g. putting away the chair when leaving the classroom).

All four forms of extrinsic motivation assume the presence of a goal. This is consistent with what was discussed in the previous sections, where the goal is a source of extrinsic motivation. However, the model presented at the beginning of this section shows that extrinsic motivation can also be driven by stimuli (situational incentives). It is then possible to identify two main categories of extrinsic motivation: goal driven and stimulus driven.

The following section will explain the origin and the implications of intrinsic motivation on behavior, especially during a learning task.

### 2.2.3 Intrinsic motivation

Intrinsic motivation is a different mechanism than extrinsic motivation. It allows for intrinsically rewarding behaviors such as exploration and curiosity, without necessarily being related to an internal or external organic stimulus.

According to François Guillemette, "motivation promotes the development of autonomy when it is intrinsic, that is to say when what pushes the learner to commit and persevere in his learning is an advantage that he found in the learning itself (intrinsic) and not in a fallout from the learning (extrinsic)" (Guillemette, 2004). It is then the object of learning and / or the act of learning itself that arouses interest. The author explains that intrinsic motivation (or curiosity) positively influences learning through its action on cognitive processes: increase in the intensity of attention, the ability to concentrate, the efficiency of memory or even courage when making a decision in the face of an unknown situation. Commitment, participation and persistence in a learning task are indicators of intrinsic motivation.
Stanislas Dehaene writes that curiosity is a force, aroused by new information, that would encourage the learner to explore. It would be the manifestation of intrinsic motivation (Dehaene, 2018). He adds that curiosity directs the learner towards new or surprising but accessible elements. That is, neither too simple to understand nor too complicated. The author cites the work of Frederic Kaplan and Pierre-Yves Oudeyer¹⁰⁸ on these aspects.

The phenomenon previously mentioned in terms of complexity and intermediate novelty sought by curiosity is called *flow*¹⁰⁹ or optimal experience. Philippe Carré cites Csikszentmihalyi who established the characteristics of flow (Carré and Fenouillet, 2019):

- "The task being undertaken is achievable, but challenging and requires a particular skill."

- "The individual is fully focused on what he or she is doing, without distraction."

- "The intended target is clear and the current activity provides immediate feedback."

- "The individual exercises control over his or her actions."

- "Preoccupation with self disappears, but, paradoxically, the sense of self is enhanced as a result of the optimal experience."

- "The perception of duration is altered."

On the other hand, Alain Lieury cites experiments that have shown that intrinsic motivation is diminished by the integration of rewards (monetary rewards, for example) into the task in the same way as constraints (supervision, social evaluation, limited time, etc.). That is, a previously rewarded activity ceases to elicit intrinsic motivation once it no longer presents a reward, whereas the same activity with no reward at all continues to elicit curiosity (Lieury, 2017). He also highlights the impact of the feeling of perceived competence (or self-esteem) on intrinsic motivation: *"this competence component was added to explain that good results generally reinforce motivation, and the opposite for poor results"* (Lieury and Léger, 2020). The need for self-determination or free will also impacts intrinsic motivation: the results of a task would then be improved when the individual is free to perform the task without supervision, without other constraints, without reward and with the feeling of being competent in solving the task.

Regarding learning, a negative self-determined feeling (feeling incompetent) favored by too high a level of difficulty, or constraints (social judgment, evaluation, etc.) can cause abandonment or resignation during learning tasks. In this situation, the learner would not be in the *flow*. Conversely, the author explains that active pedagogy, that is to say the involvement of learners in the choice of tasks and in the means to be implemented to solve them, would make it possible to promote the feeling of self-determination and intrinsic motivation.
## 2.3 The role of motivation in human learning: conclusion

In section 2, the study of the learning process allowed us to identify different theories that contribute to the understanding of what learning is and how it works. Different modes of acquisition were also cited, including problem-solving, which is used in the Crea-Cube task. It appears that these theories influence this problem-solving task in various ways and to varying degrees. In addition, setting subgoals and identifying the goal state are essential not only for the learner to solve the task but also to represent and model it.

Moreover, research conducted on motivation (in terms of learning) shows that it can be of two types: extrinsic or intrinsic. The CreaCube task integrates these two types of motivation. Indeed, the stimuli sent by the instruction and by the objects trigger extrinsic motivation with, in particular, the identification of the goal state by the learner and the setting of sub-goals that will allow him to reach this final state. On the other hand, the playfulness and autonomy left to the learner to solve this task leads to intrinsic motivation acting as a motor for his exploration and creativity.

As previously mentioned, the FLOWERS team has studied the process of intrinsic motivation in learning as well as its modeling. The researchers of this team have not only created algorithms that allow to promote the engagement of this motivation in educational tasks but also to lead the learner to remain in a state of *flow* throughout his learning so that his curiosity remains mobilized. Their work will thus be studied in the following section (section 3).

# 3 Modeling intrinsic motivation in learning

The FLOWERS (FLOWing Epigenetic Robots and Systems) research team¹¹⁰ has, among other things, been interested in the role of intrinsic motivation in learning. It has sought to develop algorithms that promote curiosity during learning by using digital applications with an educational purpose. This is what will be discussed in this section.

Furthermore, the collaboration between this team and the Mnemosyne team within the AIDE (Artificial Intelligence Devoted to Education) exploratory action¹¹¹¹¹², which aims to develop a model of the learner, will also be discussed here.

To do this, the problem-solving task (CreaCube)¹¹³ will allow us to collect different data in order to arrive at their symbolic representation in the form of an ontology. This model will take into account the motivations¹¹⁴ that lead the learner to act in his environment in order to solve the task and thus try to better understand the mechanisms that allow the learner to learn.

<footnote>110The team https://flowers.inria.fr/</footnote>

<footnote>111https://www.inria.fr/fr/aide-nouvelle-action-exploratoire-chez-inria</footnote>

<footnote>112https://team.inria.fr/mnemosyne/en/aide/</footnote>

<footnote>113https://creamaker.wordpress.com/</footnote>

<footnote>114Extrinsic motivation driven by stimuli and/or goals and intrinsic motivation driven by curiosity.</footnote>
## 3.1 The involvement of curiosity in learning

Pierre-Yves Oudeyer explains that curiosity is a form of intrinsic motivation¹¹⁵ that is fundamental to making learning more effective. It makes the learner more active in his or her learning and encourages spontaneous exploration and memorization. This is made possible by an intrinsic reward system that is activated when the learner experiences a gain in information, novelty or complexity. Csikszentmihalya's theory of flow¹¹⁶ is very important here, since it is a matter of offering the learner activities with a level of novelty¹¹⁷ and intermediate complexity to encourage curiosity. Novelty would mobilize attentional resources in such a way as to increase the duration of involvement, perseverance in the activity.

The theory of learning progress would explain what "an activity with an intermediate level of complexity" means. This theory proposes to conceive of the brain as a predictive machine intrinsically motivated to pursue activities for which the degree of uncertainty decreases as learning takes place, which would correspond to the process of improving predictions¹¹⁸. This means that the learner loses interest in the task if it is too easy or too complicated. In this case the uncertainty prediction would be either too low to generate learning, or too high without being able to be reduced (i.e., the activity would be too difficult to observe a progression in learning)¹¹⁹.

In addition, research conducted in neuroscience and psychology on curiosity postulates that there would be a unidirectional causal link between learning and curiosity. Indeed, the brain would be motivated to find an activity for which the level of novelty and complexity would be "intermediate" which would generate a state of curiosity allowing the improvement of learning and memorization¹²⁰. However, at this stage, the model shows that learning itself has no impact on the state of curiosity whereas the theory of learning progress would include a positive feedback between learning and intrinsic motivation directed by curiosity.

Figure 22 illustrates this duality. The first point of view addressed corresponds to Figure 22a, according to which the intrinsic motivation (first blue rectangle) to seek an intermediate level of novelty and complexity would lead (orange arrow) to the state of curiosity (second blue rectangle), which in turn leads to an intrinsic reward (green arrow) in *feed back*. The state of curiosity reached promotes learning and memorization (third blue

Intrinsic motivation is defined in section 2.2.3. It differs from extrinsic motivation (defined in section 2.2.2).

The theory of flow was presented in detail in section 2.2.3.

That is, the degree of similarity or distance between the perceived stimulus and the internal representations present in memory: "approach or avoidance of novelty would depend on the degree of novelty, ie, the degree of distance/similarity between the perceived stimuli and existing internal representations in the brain" (Oudeyer et al., 2016).

"This hypothesis proposes that the brain, seen as a predictive machine constantly trying to anticipate what will happen next, is intrinsically motivated to pursue activities in which predictions are improving, i.e., where uncertainty is decreasing and learning is actually happening" (Oudeyer et al., 2016).

"This means that the organism loses interest in activities that are too easy or too difficult to predict (ie, where uncertainty is low or where uncertainty is high but not reducible) and focuses specifically on learnable activities that are just beyond its current predictive capacities" (Oudeyer et al., 2016).

"The brain would be motivated to search for (intermediate) novelty or complexity, and then when finding it would be in a curiosity state that would foster learning and memory retention" (Oudeyer et al., 2016).
rectangle). There is no feedback in this process. This would imply that learning would not have an impact on curiosity or intrinsic motivation.

<figure><img src="image_23.png" /><figcaption>(a) Unidirectional causal link between learning and curiosity.</figcaption></figure>

<figure><img src="image_24.png" /><figcaption>(b) Learning progress theory: causal link with positive feedback.</figcaption></figure>

<caption>**Figure 22 - Relationship between learning and curiosity.**</caption>

This figure is made by Pierre-Yves Oudeyer (Oudeyer et al., 2016). Figure 22a is a representation of the unidirectional causal link between curiosity and learning according to research conducted in neuroscience and psychology. Figure 22b represents the same causal link but according to the learning progress theory that incorporates a positive feedback loop between learning and the curiosity state.

The theory of learning progress is illustrated in Figure 22b and shows the same causal relationship except that a positive feedback loop (blue arrows) allows us to understand that learning and memorization causally influence the state of curiosity and intrinsic motivation. This phenomenon would then allow the learner to persevere in the task and maintain a state of curiosity as long as it generates a progression in learning (that is to say as long as it presents a level of novelty and of intermediate complexity adapted to the progress of the learner in his learning). It is this progression in learning that maintains the state of curiosity, thus promoting learning itself and memorization. The feedback loop acting on intrinsic motivation makes it possible to readjust the predictions according to the errors (differences between the learning predictions previously made and the learning actually achieved).

In a context where the learner would have a choice between several activities, spontaneous active exploration (according to the learning progress theory) will direct him towards the activity which, according to his predictions, will allow maximum progress in learning by avoiding activities judged too easy or too difficult. The learner will persevere in this activity until there is no more progress in learning. He will then resume an exploration phase in search of another activity that will again allow him to achieve maximum progress.

In order to understand precisely what is at stake for the learner and his environment during this exploration phase, it is necessary to give a definition of it. According to Jacqueline Gottlieb, exploration is a process consisting of choosing actions whose purpose
is to obtain information. Actions with an exploratory purpose are different from physical
actions. These exploratory actions may, however, involve physical actions. The difference
between these two types of actions is that physical actions are intended to change the
learner's external or internal environment, to interact with the learner physically (e.g.,
grabbing an object). Exploratory actions do not seek to impact the environment but aim to
gather information about that environment and change the learner's epistemic state¹²¹.

Furthermore, Frédéric Alexandre [Alexandre, 2021] specifies that the learner explores in order to obtain new information, but that he can do so in two ways: either by following a strategy, or randomly (when there is no other solution). At the same time, the author explains that exploration can be done by internal simulations (*covertly*) or by real actions (*overtly*). The choice between these two forms of exploration is made by taking into account the cost that they generate and their feasibility. Simulation can be less costly and will therefore be preferred. However, its implementation may be impossible in some cases, so real actions will be preferred. On the other hand, real actions can also be less costly from the start, which will rule out the choice of simulation.

These explanations of the role of exploration in learning, and more specifically in
learning progress, complement the work of Joy Paul Guilford cited in section 1.3.4 regarding
his involvement in creativity, in section 2.1.1 where Jennifer Kerzil makes the link with
constructivist learning theory, or in section 2.2.3 where François Guillemette defines intrinsic
motivation.

Finally, Pierre-Yves Oudeyer and Jacqueline Gottlieb explain that computational
models based on a mathematical formalism have been developed to study curiosity-driven
reinforcement learning. Some of these models will be presented in section 3.2.

## 3.2 Computational models of curiosity

### 3.2.1 Intelligent Tutoring Systems

Benjamin Clément and his collaborators from the FLOWERS team have been working on Intelligent Tutoring Systems (ITS)¹²² and have developed algorithms that improve learning performance by acting on learner motivation through the use of digital activities, as observed experimentally. The author explains that these tutoring systems allow for a more accessible, more effective education and at the same time collect objective measures on learning¹²³. They consist of four main models:

*   **Cognitive model.** It defines the domain of knowledge or the steps to be taken to solve problems in a particular domain¹²⁴.

121"Multiple paradigms have been devoted to the study of exploration and have used a common definition of this process as the choice of actions with the goal of obtaining information. Although exploratory actions can involve physical acts, they are distinct from other motor acts in that their primary goal is not to exert force on the world, but to alter the observer's epistemic state. For instance, when we turn to look at a new storefront, the goal of the orienting action is not to affect a change in the external world (as we would, for instance, when we reach for and grasp an apple). Instead, the goal is to obtain information" (Gottlieb et al., 2013).

122Literally, "Intelligent Tutoring Systems".

123Intelligent Tutoring Systems (ITS) have been proposed to make education more accessible, more effective and simultaneously as a way to provide useful objective metrics on learning" (Clément et al., 2014a).

124"A cognitive model that defines the domain knowledge or which steps need to be made to solve
* **Learner model.** It considers how the learner learns, the evolution of his cognitive state during particular learning activities¹²⁵.

* **Learning model.** It defines the activities to be presented to learners based, in a general way, on the cognitive model and the learner model¹²⁶.

* **User interface model.** It allows to represent the way interactions with the learner take place and how problems are proposed to them¹²⁷.

The authors aim to create an intelligent tutoring system by working at the level of the learning model in order to identify the sequence of activities that would maximize the learner's average skill level for the set of skills being worked on. The work focuses on how to select the activities that would provide the best learning experience based on an estimate of the learner's skill level, progression, and knowledge from the cognitive model and the learner model. The researchers want to offer each learner the activities that will provide the best learning progress.

The approach consists in taking into account the activities estimated to bring the maximum learning gain, this at the execution of the activity and based on the learner's results. Three main advantages have been identified:

* **Lower dependence on the cognitive/learner model.** The independence of the learning model (as independent as possible from the cognitive/learner model¹²⁸) requires that the *intelligent tutoring system* explore and experiment with various activities to estimate their potential for learning progress for each learner. The experiments should be sufficiently informative about the student's current skill level as well as estimate the effectiveness of each exercise in improving those skills. This has been called the "exploration/exploitation" trade-off in machine learning, where new activities must not only be tried to find out which are the best, but also simultaneously select the best ones for the learner to actually learn¹²⁹.

* **Effective optimization methods.** These methods make no specific assumptions about how learners learn and only require information about the estimated learning progress of each activity. The author makes the assumption that activities that are

problems in a particular domain" (Clémentet al., 2014a).

125"A student model that considers how students learn, what is the evolution of their cognitive state depending on particular teaching activities" (Clémentet al., 2014a).

126"A tutoring model that defines, based on the cognitive and the student model, what teaching activities to present to students" (Clémentet al., 2014a).

127"A user interface model that represents how the interaction with the students occurs and how problems are proposed to the learners" (Clémentet al., 2014a).

128Both of these models require too much data on the learner and the activities to be feasible for a class of students for example. They take into account the particularities and difficulties of each learner. The analysis of these particularities must make it possible to identify which activity will bring the maximum level of learning progress. This implies taking individual responsibility for each learner.

129"The technical challenge is that these experiments must be sufficiently informative about the student's current competence level and also to estimate the effectiveness of each exercise to improve those competences (a form of stealth assessment). This boils down to what has been called the "exploration/exploitation" trade-off in machine learning, where we have to simultaneously try new activities to know which ones are the best, but also select the best ones so that the student actually learns" (Clément et al., 2014a).
estimated to provide a good learning gain over the course of the activity should be
selected more often. He cites the *Multi-Armed Bandits* formalism¹³⁰¹³¹.

* **More motivating experience.** This refers to the zone of proximal development¹³² which seeks to foster the emergence of the state of flow¹³³ in order to enhance learning.

### 3.2.2 Contextualization of the algorithms

Their work has included the development of two machine learning algorithms
(specifically, reinforcement learning):

* **RiARiT (Right Activity at Right Time)¹³⁴** which explicitly estimates the learner's level of various components of competence (*Knowledge Components*¹³⁵ : KC). The algorithm uses these estimates to propose activities to the learner¹³⁶.

* **ZPDES (Zone of Proximal Development and Empirical Success)¹³⁷** which uses a graph and follows the learner's progress to choose the activities to propose¹³⁸.

In both cases, a **Pedagogical Activity Space** is defined beforehand. It is a set of activities that the learner can carry out to acquire competencies or components of competency¹³⁹.

To design their algorithms, the authors were inspired by a Multi-Armed Bandit¹⁴⁰¹⁴¹ EXP4 (Exponential Weights for Exploration and Exploitation¹⁴²) algorithm. This variant of the Multi-Armed Bandit algorithm helps to automatically determine, by random draw¹⁴³, the share

130This formalism will be discussed below.

131"We will rely on methods that do not make any specific assumptions about how students learn and only require information about the estimated learning progress of each activity. We make a simple assumption that activities that are currently estimated to provide a good learning gain, must be selected more often. A very efficient and well studied formalism for these kinds of problems is Multi-Armed Bandits" (Clément et al., 2014a).

132This was discussed in section 3.1 and corresponds to the intrinsic motivation of the learner when faced with an activity that is neither too easy nor too difficult but just beyond his or her capabilities, as presented in the learning progress theory.

133See section 3.1.

134The right activity at the right time.

135This notion refers to elementary competences involving cognizances and know-how, which are called knowledge components by the authors. It is characterized by the term "composantes de compétence" in the document in French

136"*The first one is RiARiT, which explicitly estimates the level of the student's proficiency for different KCs to base its choice of activities*" (Clément, 2018).

137Zone of proximal development and empirical success.

138"*The second is ZPDES, which uses a "graph" of activities and tracks progress to choose the next activity; it does not use a student model based on KCs*" (Clément, 2018).

139"*A pedagogical Activity Space is considered to be a set of activities that a learner can practice to acquire skills or knowledge components*" (Clément, 2018).

140"Bandit manchot" algorithm is used in particular to choose the slot machine which will maximize the player's winnings according to probabilities.

141Multi-Armed Bandits is a set of different techniques but based on the same principle. As the details of these techniques will not be discussed in this work, the term "Multi-Armed Bandit algorithm" will be used in order to simplify the discussion.

142Exponential weights for exploration and exploitation.

143The notion of random choice is important in the difference between exploitation and exploration. Exploration is based more on random choice than exploitation, which is based more on the use of prior knowledge.
of exploitation and exploration at a time, i.e., whether the player should continue using a given machine or try others at that time<fnref n="144" /> <fnref n="145" /> <fnref n="146" />.

From the point of view of learning, and in the context studied by the authors, their algorithms make it possible to propose to the learner a sequence of activities which has the greatest probability of maximizing learning progress and consequently of allowing him to preserve the state of flow<fnref n="147" /> during the learning session.

Thus, the Multi-Armed Bandit mechanism is used in these algorithms, except that it is no longer slot machine arms that are recommended but human learning activities, and this with the help of machine learning algorithms by reinforcement<fnref n="148" />. The gain of money is replaced by the learner's progress in learning.

To link this to the approach identified in the previous section (3.2.1), it is important to remember that the authors have chosen to favor the learning model by minimizing as much as possible, in particular, the dependence on the cognitive/learner models. This implies that no (or little) information concerning the learner's skills and particularities is given as input. At the same time, the information concerning the activities is also limited. For this, the pedagogical expert (e.g. a teacher) defines the activity space and, if necessary, some roughly defined constraints on the pedagogical sequence<fnref n="149" />.

The ZPDES algorithm respects this principle since it has no a priori information on the learner's skills and very little on the problem<fnref n="150" />. It uses the zone of proximal development and empirically estimates<fnref n="151" /> learning progress<fnref n="152" />.

The RiARiT algorithm, on the other hand, is much more dependent on cognitive/learner models, since it uses a table of data on expected levels of competence for each activity parameter, in order to relate the learner's competences to the activity parameters. His estimates of learning progress are directly related to this relationship<fnref n="153" />.

To go further, and in the hypothesis that these algorithms could participate in the modeling of the learner during a problem solving task generating a learning gain, it is possible to establish a link with the CreaCube task. Indeed, the modeled learner must perform a certain number of actions to solve the task. Here, the algorithm would not propose an activity but an action to perform. The notions of exploitation and exploration remain true in the sense that the learner can explore the cubes totally at random, explore in a more directed way (implication of the notion of probability while preserving a part of randomness)

<footnote n="144">144 http://www.math.univ-toulouse.fr/~jlouedec/demoBandits.html</footnote>

<footnote n="145">145 https://fr.wikipedia.org/wiki/Bandit_manchot_(math%C3%A9matiques)</footnote>

<footnote n="146">146 https://hal.inria.fr/tel-01420663v1/document</footnote>

<footnote n="147">147 As a reminder: "theories of intrinsic motivation clearly suggest motivation and learning improve if exercises are proposed at levels that are only slightly higher than the current level" (Clément et al., 2015).</footnote>

<footnote n="148">148 Assigning rewards based on the match between the prediction and the outcome.</footnote>

<footnote n="149">149 "Our approaches assume that an instructional expert defines a set of skills to acquire, a set of potential activities/exercises to practice and, if necessary, coarse constraints on the pedagogical sequence" (Clément et al., 2015).</footnote>

<footnote n="150">150 See section 3.2.3.</footnote>

<footnote n="151">151 Estimates are based on observation of the learner's success or failure in previous activities.</footnote>

<footnote n="152">152 "Our first approach uses very little knowledge about the problem and is inspired by the zone of proximal development and the empirical estimation of learning progress hence the name "Zone of Proximal Development and Empirical Success" (ZPDES)" (Clément et al., 2015).</footnote>

<footnote n="153">153 "Our second approach further assumes the existence of a simple relation between the activities and the skills. Then, at any given point in time, the system estimates the learning progress obtained for each activity by the student. The system then proposes to the student the activities which provide a higher learning progress, hence the name of the algorithm : the "Right Activity at the Right Time"(RiARiT)" (Clément et al., 2015).</footnote>
or exploit the acquired competencies and/or cognizances. This can be further clarified with the study of algorithms in the next section (3.2.3).

### 3.2.3 Presentation of algorithms

The efficiency of the algorithms depends on several elements. First of all, an expert (or several) must define the activity space in which the learner will evolve. Then, since these algorithms are based on a *Multi-Armed Bandit* (MAB) type algorithm, it is necessary to understand the main principles before being able to study them more precisely.

#### Activity space

Each activity in the activity space is characterized by parameters (e.g. type of activity) that can take different values (e.g. type A or type B). These parameters and their respective values define all the possible activities that can be instantiated within this space<fnref n="154" />. These parameters can be organized into groups according to their nature and meaning. The groups can be structured hierarchically since some parameters depend on other parameters in order to be used in an activity<fnref n="155" />. Furthermore, not all groups of parameters are necessarily used to define all activities in the space.

An activity is characterized by a combination of parameter values selected in each group of the activity space (necessary for its characterization) in respect of the hierarchy. An activity space groups all possible distinct combinations of parameter values that can define an activity in this space<fnref n="156" />.

Figure 23 is an example of the process of selecting an activity in an activity space. In this space, group 1 is the first in the hierarchy. It groups the "type of activity" parameters. This group is composed of a single parameter (T) with 2 values (1 and 2). The activity can therefore be of type T1 or T2. Following the hierarchy, each type T of activity must pass through a particular group of difficulty (group 2 for T1 or group 3 for T2). Each of these groups has a parameter (level) with three values (1, 2 and 3). Then, group 4 is the last one of the hierarchy and allows the attribution of modalities. This group has two parameters (shape and color) with respectively 2 (square and circle) and 3 values (Black, Blue and Red). The combination of the example leads to the proposal of an activity of type T2 (group 1), of level 1 difficulty (group 3) and has two modalities (group 4): square, blue.

The algorithms mentioned above will allow efficient use of this activity space in order to offer pertinent, personalized and motivating activities for the learner. They will be studied in more detail in the following paragraphs.

#### Multi-Armed Bandit

The Multi-Armed Bandit EXP4 algorithm consists, globally, in assigning an initial weight to each action, which quantifies the interest of this action in the current task. An action is then drawn according to both the weights and chance, in relation to a probability of

<footnote n="154">154 "These parameters and their respective values define all the possible activities that can be instantiated inside the activity space" (Clément, 2018).</footnote>

<footnote n="155">155 "Depending on their nature and meaning, these parameters can be organized in different groups. [...] In addition, these parameter groups can be structured hierarchically, since some parameters depend on others to be used in an activity" (Clément, 2018).</footnote>

<footnote n="156">156 "An activity space groups all possible distinct combinations of parameter values that can define an activity in this space" (Clément, 2018).</footnote>
gaining. The probability of gaining is estimated thanks to the exploration/exploitation trade-off. This trade-off requires making a choice between what is known in order to obtain a reward close to what is expected (exploitation), and what is uncertain in order to possibly learn more (exploration)¹⁵⁷.

Initially, the exploration part is stronger since no reward is known. It is the observation of the reward obtained that makes possible to update the weights attributed and consequently the probabilities of gain. Thus, the activities giving a good level of learning progress (reward) must be those selected most often. Since learning progress is not a stable reward, it requires specific mechanisms to follow its evolution. Indeed, an activity can cease to provide learning progress once the learner has reached a certain level of expertise in the skill being worked on or in the activity itself¹⁵⁸. This implies that the rewards differ from one learner to another. This is why the EXP4 variant of the Multi-Armed Bandit algorithm was selected by Benjamin Clément and his collaborators.

A set of "expert agents"¹⁵⁹ is then taken into account and the model makes a choice based on the proposals of each internal expert agent. In the case presented here, the experts are a set of variables that follow the evolution of the reward provided by each activity. These "expert bandits" are used to evaluate the quality of each activity parameter value during the learner's work session¹⁶⁰.

Thus, Benjamin Clément explains that, for each parameter within a group, the quality of its values is evaluated by an "agent-expert bandit". An expert follows the reward provided by each value over the last few samplings to compute its quality. At a given time, the value to use for each parameter is sampled according to the probabilities. This sampling methodology leads to selecting stochastically (by chance) a value, proportionally to its quality and its exploration rate. When the exploration rate is low, the value of the parameter is chosen according to its quality. Conversely, when the exploration rate is high, low quality parameter values have a higher probability of being selected. To generate an activity, this process is performed in a recursive manner on the groups involved in the activity generation, according to the hierarchical dependencies between the parameter groups¹⁶¹. This leads to a stochastic drawing of the activity, resulting from the combination of each sampled parameter value according to the assessment of their quality by each agent-expert (Clément, 2018)].

The selected activity is proposed to the learner for resolution. The reward level

<footnote>157"*The exploration-exploitation trade-off is a fundamental dilemma whenever you learn about the world by trying things out. The dilemma is between choosing what you know and getting something close to what you expect ("exploitation") and choosing something you aren't sure about and possibly learning more ("exploration")*" (Clément, 2018).</footnote>

<footnote>158“A particularity here is the reward (learning progress) which is non-stationary. This requires specific mechanisms to track its evolution. Indeed, a given activity will stop providing a reward, or learning progress, after the student reaches a certain mastery level of the skill or of the activity” (Clément, 2018).</footnote>

<footnote>159The general term "expert" is borrowed from Cesa-Bianchi et al (1997). They use it to refer to the strategies used in "expert-advised prediction" algorithms, "combining the predictions of several prediction strategies." (Clement, 2018).</footnote>

<footnote>160"*Thus, the framework introduced here rely on a variant of the EXP4 algorithm, proposed initially by (Auer et al., 2003), which considers a set of experts and make a choice based on the proposals of each expert. In case presented here, the experts are a set of variables that track how much reward each activity is providing (Lopes and Oudeyer, 2012). These bandit experts are used to evaluate the quality of each activity parameter value during the learner's working session*" (Clément, 2018).</footnote>

<footnote>161See section 3.2.3, paragraph "Activity Space" and figure 23.</footnote>
obtained (learning progress) allows the expert-agents to update the quality of each
parameter value used. All this would cause the exploration of all the activities generated in
the activity space. This leads the author (Clement, 2018) to identify two resulting drawbacks:

* The type and difficulty of the exercises offered could change too often and reduce learner motivation and engagement.

* It may not be possible to explore all activity parameters to estimate their learning progress.

In order to overcome these drawbacks, the author was inspired by the theory of the *Zone of Proximal Development* (ZPD) and the concept of *flow*¹⁶². It is then a pedagogical expert (or several) who defines the activity space and, consequently, the hierarchy of activities within it as well as the parameters/value pairs. The aim is to propose only the most pertinent activities and to keep the learner in the zone of proximal development or in the *flow* zone.

The level of difficulty of the activities related to the skill level of the learner determines what state the latter is in during learning, as shown in Figure 24. Thus a level of difficulty that is too high in relation to the level of competence of the learner will generate a state of anxiety. Conversely, an insufficient level of difficulty compared to the skill level will cause a state of boredom. The state of *flow* is reached when the skill level and the level of difficulty are almost proportional. The zone of proximal development is one level above, that is, the proposed activity is slightly above the learner's current abilities, allowing the maximum level of learning progress to be generated while maintaining learner motivation and engagement. Furthermore, the use of the zone of proximal development makes it possible to reduce the need for quantitative measurements initially necessary for the expert and makes it possible to provide a more predictive choice of activity¹⁶³.

## The ZPDES algorithm

The aim of the ZPDES algorithm is to propose to the learner the activities that allow for maximum *learning progress* while reducing as much as possible the dependence of the model on the cognitive/learner models¹⁶⁴. It is the zone of proximal development established by a pedagogical expert that makes it possible to define the set of activities that can be proposed in a way that is consistent with the learner's estimated level of competence. Benjamin Clément and his collaborators have represented this in the form of a graph as shown in Figure 25.

In the example shown in Figure 25, the activity graph is represented in three states. It shows three types of activity (orange disks) of varying difficulty. The difficulty of the activities

<footnote>162"Inspired by the Zone of Proximal Development theory (Vygotsky, 1930-1934/1978) and the concept of Flow (M. Csikszentmihalyi and I. Csikszentmihalyi, 1975), a pedagogical expert has the possibility to specify rules that define an evolving set of possible/activated activities, judged relevant for the student. These activities keep the student in the zone of Flow or in the Zone of Proximal Development (ZPD) based on his successive results" (Clément, 2018).</footnote>

<footnote>163"The use of the ZPD offers three advantages : it helps to improve motivation as discussed before, it further reduces the need of quantitative metrics for the educational design expert and it provides a more predictive choice of activities" (Clément, 2018).</footnote>

<footnote>164"Two sources of inspiration are used to simplify the algorithm [RiARiT]: Zone of Proximal Development (C. D. Lee, 2005) and the empirical estimation of learning progress (Oudeyer and Kaplan, 2007)" (Clément, 2018).</footnote>
increases according to their type (C is more difficult than B which is more difficult than A). A type groups several activities of increasing level (A3 is more difficult than A2 which is more difficult than A1).

The initial state (Figure 25a) shows that Activity 1, type A, was proposed to the learner (blue circle). It is located in the zone of proximal development (yellow zone) and has the lowest level of difficulty.

In Figure 25b, the learner has completed the activity and passed (green circle). The algorithm then proposes another activity in the zone of proximal development that has evolved thanks to the update of the learner's progress level. The level of progress has been updated according to the result obtained in the first activity. The second proposed activity (blue circle) is of the same type but of a higher level of difficulty.

Figure 25c shows the next state of the graph. Here, the learner has not succeeded in the previously proposed activity (black circle). The zone of proximal development has been adapted again, taking into account the level of progress. Since the A2 activity was not successful, intermediate level activities were added to the zone and the higher level activity was removed (A3). Therefore, Activity B1 was proposed.

The activities can be proposed with a certain amount of randomness among the activities of the zone of proximal development likely to maximize the level of progress, that is to say the state (success or failure) of the activities previously performed (the closest). Thus, in graph 25c, if the learner succeeds in activity B1, it is possible that activity B2 or C1 will be proposed to him according to the evolution of the zone of proximal development which is defined according to the level of difficulty of the activities in the graph and the state of success of the previous activities.

To summarize, one (or more) pedagogical experts create the activity space represented as a graph. The activities are distributed in the graph according to their type and level of difficulty, as in figure 25. The graph is traversed according to the learner's results and the evolution of the zone of proximal development.

The zone of proximal development is established by taking into account the hierarchy of the activity space graph in order to maintain the learning progression. The zone encompasses the activities of the graph with a higher level of difficulty than the activity performed by the learner¹⁶⁵. The *learning progress* is the reward of the algorithm¹⁶⁶.

Once the zone of proximal development is established, the algorithm stochastically proposes an activity contained in the zone. That is to say that the choice of the activity depends on probabilities calculated with a random part (exploration) while taking into account the level of competence (exploitation). This is the exploration/exploitation compromise.

The learner performs the proposed activity. His or her success or failure allows the reward to be updated, i.e., the level of progress that also takes into account the results of the

165It is the activities closest to the activity performed that constitute the zone of proximal development in order to offer slightly more difficult activities to promote learning progression.

166Since it is reinforcement-based machine learning, it adapts its predictions according to the level of reward it receives.
previous activities. When progress stagnates, the zone of proximal development evolves to propose more difficult activities. Similarly, if the reward decreases, the zone of proximal development is re-evaluated, thus proposing another type of activity of the same level or a lower level activity.

## The RiARiT algorithm

The RiARiT algorithm is heavily informed about the activity domain and the learner, which makes it quite dependent on the learner's model. The information needed by the algorithm is used to design the cognitive model by estimating the learner's skill level based on their responses to each activity and the definition of a set of skill components for each activity¹⁶⁷. The resulting model is also used to tailor the zone of proximal development.

RiARiT takes a table as input containing the skill components worked on in the activity space. This activity space is itself characterized by activity parameters and values. The author uses what are called *q-values*¹⁶⁸. The *q-values* (*quality values*) are continuous numbers between 0 and 1, where 1 means that the value of the parameter provides the entire level of competence that it is possible to acquire with this parameter for the competence component worked on. Conversely, a *q-value* equal to 0 indicates that the value of the parameter does not allow the acquisition of this competence component.

The algorithm takes into account these components and parameters, set by one (or more) expert, to establish the zone of proximal development. To do this, it estimates the learner's skill level and adjusts the zone according to the difference between the expected level (set in the table) and the estimated level in order to propose the most relevant activity at a given time *t*. It is this difference that constitutes the algorithm's reward for each component.

Specifically, if the estimated skill level is lower than the expected skill level, then the learner's level was underestimated (the activity was too easy), resulting in a positive reward. Conversely, if the estimated skill level is higher than the expected skill level, then the learner's skill level was overestimated (the activity was too difficult), resulting in a negative reward.

When the learner gives a **correct answer** and the **reward is positive** then the estimated skill level is updated. Similarly, when the learner gives a **wrong answer** and the **reward is negative**, the estimated skill level is also updated. However, in other cases, the estimated skill level is kept until the next update.

The area of proximal development is also updated based on the new estimated competence value.

Alternatively, the evolution of the zone of proximal development can be based on explicit values of the learner's level of competence. The pedagogical expert can specify the

<footnote>167"It is strongly informed about the activities domain and the student. This information will be used to build a cognitive model of the student by estimating the knowledge level of the learner depending on his answer to each activity and the definition of a set of Knowledge Component (KC) related to each activity" (Clément, 2018).</footnote>

<footnote>168"The *q-values* notion take its inspiration from the work of Barnes (2005) where she defines a *Q-matrix* or "attribute by item incidence matrix" to define relations between questions (activities in our case) and concepts (KCs in our case) and contains a one if a question is related to a concept and a zero if not" (Clément, 2018).</footnote>
minimum level of competence required in a given competence component. Thus, each
parameter value will be explored only if the learner's skill level is above the threshold value.
Pedagogical experts are allowed to define the threshold for which a given parameter is
removed from exploration, its agent-expert bandit is then deactivated¹⁶⁹.

This is consistent with theories of intrinsic motivation, which clearly suggest that
motivation and learning increase if the level of activities offered is just a little above the
current level¹⁷⁰.

In conclusion, this algorithm uses a lot of information about the domain of activity.
The pedagogical expert defines tables with the relationship between parameter values and
competency components, as well as a set of minimum competency levels to activate a new
parameter value. The relationship between the success of an activity, the estimated skill
level, and the required skill level of an activity allows two things:

* Estimate the level of the learner.

* Calculate rewards for this activity¹⁷¹.

The information required for this algorithm is more computed over time than many
other Intelligent Tutoring Systems, as this a priori information can be very difficult for a
pedagogical expert to give when the number of activities, or competence components, is
high. Automatic methods for filling in this knowledge already exist and are being actively
researched¹⁷².

## Comparison of the RiARIT and ZPDES algorithms

Experiments have been conducted to test the effectiveness of these algorithms (Clémentet al., 2014b) (Clément et al., 2014a) (Clément et al., 2015) (Clément, 2018). It appears that learners make more and faster progress using these algorithms. The RiARit algorithm being based on a priori data is somewhat more efficient than ZPDES. However, it is not easily transferable to other domains with more data because the inventory of components and parameters would be too large to be feasible. The difference in efficiency with ZPDES, in terms of learning progress, is small enough to favor the latter, especially for

169"Here the evolution of the ZPD can rely on explicit values of the estimated competence level of the learner. Thus, the expert can specify minimal competence levels in given KC that are required to allow the algorithm to try a given parameter values and activate its related bandit expert. Each parameter value is only explored if the learner is already above this minimum threshold. The teaching experts are allowed to define a threshold for which a given parameter is removed from the exploration, its bandit expert is deactivated" (Clément, 2018).

170"This follows well known instructional design methodologies (Gagne and Briggs, 1974) and concords with theories of intrinsic motivation which clearly suggest motivation and learning improve if exercises are proposed at levels that are only slightly higher than the current level (Engeser and Rheinberg, 2008; Habgood and Ainsworth, 2011)" (Clément, 2018).

171"RiARIT algorithm uses a lot of information about the domain. The teaching expert defines tables with the relation between the parameter values and the KC, and also a set of minimum competence levels to activate a new parameter value. The relation between the success of an exercise, the estimated competence level and the required competence level of an exercise allows two things : a) to estimate the level of the student ; and b) to compute a reward for that activity" (Clément, 2018).

172"The information required for this algorithm is more online than a lot of other ITS systems but this amount of information might be really difficult to give for a teaching expert when the number of activities, or KC, is high. Automatic methods to fill such knowledge already exist and is an area of active research (Baker et al., 2008; Dhanani et al., 2014; González-Brenes, Huang, et al., 2014; González-Brenes and Mostow, 2012)" (Clément, 2018).
domains with too much or insufficiently defined starting data.

Moreover, the use of these algorithms as *Intelligent Tutoring Systems* has not only promoted learning progress but also maintained the state of *flow* in learners [Oudeyer et al., 2016]. On the other hand, they do not allow the identification of the cause (environment, tool, learner's peculiarities, etc.) and the subject (a competence component or an activity parameter, etc.) of the errors made during the activity sequence. This implies, on the one hand, the possibility of using machine learning to arouse the curiosity and intrinsic motivation of learners and thus promote progress and, on the other hand, to supervise these activities in a human way so as to identify the difficulties of each learner in order to work on them differently or to act on the learning conditions.

## 3.3 Modeling intrinsic motivation in learning: conclusion

This section has highlighted the role of intrinsic motivation in learning. The work of the FLOWERS team has allowed us to understand the mechanisms of *Intelligent Tutoring Systems* and the role that digital technology can play in improving learning by taking into account intrinsic motivation.

The algorithms of the FLOWERS team and the experiments conducted show the importance of adding the motivational dimension to the ontological model of the learner initiated within the exploratory action AIDE.

## References

Abgrall, J.P. (2012). *Stimuler la mémoire et la motivation des élèves. esf éditeur*.

Alexandre, F. (2021). A global framework for a systemic view of brain modeling. *Brain Informatics*. https://hal.archives-ouvertes.fr/hal-03143843

Besnier, J.-M. (2021). *Les théories de la connaissance*. Presses Universitaires de France. https://www.cairn.info/les-theories-de-la-connaissance--9782715406032.htm

Breedlove, S. M., Rosenzweig, M. R., & Watson, N. V. (2012). *Psychobiologie. De ma biologie du neurone aux neurosciences comportementales, cognitives et cliniques*. de boeck.

Brouillard, P. (2021). *Apprentissage de modèles causaux par réseaux de neurones artificiels*. Université de Montréal. https://papyrus.bib.umontreal.ca/xmlui/handle/1866/25096

Burglen, F. (2005). Etudes du mécanisme de binding en mémoire de travail et de la boucle phonologique chez le patient schizophrène. *Université Louis Pasteur, Strasbourg*.
Carré, P., & Fenouillet, F. (2019). *Traité de psychologie de la motivation*. Dunod.

https://www.cairn.info/traite-de-psychologie-de-la-motivation--9782100783045.htm

Censabella, S. (2007). In *Bilan neuropsychologique de l'enfant*. Mardaga.

https://www.cairn.info/bilan-neuropsychologique-de-l-enfant--9782870099643-page-117.htm

Chang, C.-Y., Tijus, C., & Zibetti, E. (2015). Les apprentissages à l'heure des technologies cognitives numériques. In *Administration & éducation*.

https://www.cairn.info/revue-administration-et-education-2015-2-page-91.htm?

contenu=article

Chappell, T. (2019). *Plato on knowledge in the Theaetetus*. Stanford Encyclopedia of Philosophy.

Chaudet, B. (2009). *Donnée, information, connaissance*.

https://brunochaudet.wordpress.com/2009/03/30/donnee-information-connaissance/

Chekour, M., Mohammed, L., & Janati-Idrissi, R. (2015). L'évolution des théories de l'apprentissage à l'ère du numérique. *Association EPI*.

https://www.epi.asso.fr/revue/articles/a1502b.htm#BPAGE

Chi, M. T., & Wylie, R. (2014). The icap framework :Linking cognitive engagement to active learning outcomes. *Educational Psychologist*, 49(4).

Cieutat, P., & Connac, S. (2017). *Constructivisme ou enseignement explicite ? Les cahiers pédagogique - Cercle de recherche et d'action pédagogiques*.

https://www.cahiers-pedagogiques.com/constructivisme-ou-enseignement-explicite/

Clément, B. (2018). Adaptive personalization of pedagogical sequences using machine learning. *Artificial Intelligence* [cs.AI]. Université de Bordeaux.

https://hal.inria.fr/tel-01968241/file/CLEMENT_BENJAMIN_2018.pdf

Clément, B., Roy, D., Oudeyer, P.-Y., & Lopes, M. (2014a). Developmental learning for intelligent tutoring systems. *IEEE ICDL-Epirob - The Fourth Joint IEEE International Conference on Development and Learning and on Epigenetic Robotics*.

https://hal.inria.fr/hal-01061195v2

Clément, B., Roy, D., Oudeyer, P.-Y., & Lopes, M. (2014b). Online optimization of
teaching sequences with multi-armed bandits. 7th International Conference on
Educational Data Mining. https://hal.inria.fr/hal-01016428

Clément, B., Roy, D., Oudeyer, P.-Y., & Lopes, M. (2015). Multi-armed bandits for
intelligent tutoring systems. Journal of Educational Data Mining, 7(2).
https://hal.inria.fr/hal-00913669

Conein, B. (2004). Cognition distribuée, groupe social et technologie cognitive. In *Réseaux*. https://www.cairn.info/revue-reseaux1-2004-2-page-53.htm

Connac, S. (2018). Neuroéducation et pédagogie. *Éducation et socialisation. Les Cahiers du CERFEE*, **49. https://journals.openedition.org/edso/3556**

Damasio, A. R. (1998). motion in the perspective of an integrated nervous system. Brain Research Reviews, 26.

https://edisciplinas.usp.br/pluginfile.php/5706589/mod_resource/content/1/emotion-in-the-perspective-of-an-integrated-nervous-system.pdf

Dehaene, S. (2018). *Apprendre ! Les talents du cerveau, le défi des machines*. Odile Jacob.

Delacour, J. (2001). *Conscience et cerveau : La nouvelle frontière des neurosciences*. Neurosciences & cognition. https://www.cairn.info/conscience-et-cerveau--9782804137663-page-5.htm

Dietrich, A. (2004). The cognitive neuroscience of creativity. *Psychonomic Bulletin & Review*, **11**. https://doi.org/10.3758/BF03196731

Donoso, M. (2013). Le cerveau stratège : les fondements du raisonnement dans le cortex préfrontal humain. *Université Pierre et Marie Curie, Paris* 6.
http://www.theses.fr/2013PA066535

Duplàa, E., & Talaat, N. (2011). Connectivisme et formation en ligne. In *Distances et savoirs* (Vol. 9). https://www.cairn.info/revue-distances-et-savoirs-2011-4-page-541.htm

Evans, J. (2011). Dual-process theories of reasoning : Contemporary issues and developmental applications. *Developmental Review*, **31**(2-3).

Faugloire, E. (2007). Approche dynamique de l'apprentissage de coordinations
posturales. Université Paris Sud - Paris XI. https://tel.archives-ouvertes.fr/tel-00128845/document

Fenouillet, F. (2016). *Les théories de la motivation*. Dunod.

Frayssinhes, J. (2019). Compétence, expérience, connaissances et savoirs transférables. https://hal.archives-ouvertes.fr/hal-02939062/document

Frayssinhes, J., & Pasquier, F. (2018). Neurosciences et apprentissages via les réseaux numériques. *Éducation et socialisation. Les Cahiers du CERFEE*, 49.
https://journals.openedition.org/edso/3920

Garnier, C., & Sauvé, L. (1999). Apport de la théorie des représentations sociales à l'éducation relative à l'environnement - conditions pour un design de recherche. *CIRADE, université du Quebec*.

Gasc, M. (2016). Étude de la fonction symbolique chez des enfants de 3 à 5 ans présentant un retard de parole et/ou de langage. http://thesesante.ups-tlse.fr/1756/1/3405A201611.pdf

George, C. (1997). *Polymorphisme du raisonnement humain*. Presses Universitaires de France.

Godin, C. (2004). *Dictionnaire de philosophie*. Paris Fayard.

Goldman, S. (2005). *La croyance : aux confins mystérieux de la cognition* (25th ed., Vol. 2). https://www.cairn.info/revue-cahiers-de-psychologie-clinique-2005-2-page-87.htm

Gottlieb, J., Oudeyer, P.-Y., Lopes, M., & Baranes, A. (2013). Information-seeking, curiosity, and attention : computational and neural mechanisms. *Trends in Cognitive Sciences*. http://www.pyoudeyer.com/TICSCuriosity2013.pdf

Goutte, V., & Ergis, A.-M. (2011). Traitement des émotions dans les pathologies neurodégénératives : une revue de la littérature. In *Revue de neuropsychologie* (Vol. 3). https://www.cairn.info/revue-de-neuropsychologie-2011-3-page-161.htm?
contenu=article

Guilford, J. P. (1956). *Structure of intellect*. Psychological Bulletin.

Guillemette, F. (2004). Favoriser l'apprentissage en favorisant la motivation
intrinsèque. Observatoire de la pédagogie en enseignement supérieur.

https://oraprCNT.uqtr.uquebec.ca/Gsc/Portail-ressources-enseignement-
sup/documents/PDF/Motivation_intrinseque_motivation_extrinseque.pdf

Habib, M., Lavergne, L., & Caparos, S. (2018). *Psycho-logie cognitive*. Armand Colin.

Han, J., Shi, F., Chen, L., & Childs, P.R. (2018). A computational tool for creative idea generation based on analogical reasoning and ontology. In *Artificial Intelligence for Engineering Design Analysis and Manufacturing*.

Hart, J. T. (1965). Memory and the feeling-of-knowing experience. *J Educat Psychol*, 56(4), 208-216. https://psycnet.apa.org/record/1965-13971-001

Hoareau, V. (2018). Etudes des mécanismes de maintien en mémoire de travail chez les personnes jeunes et âgées : approches computationnelle et comportementale basées sur les modèles tbrs et sob-cs. *Université Grenoble Alpes*. https://tel.archives-ouvertes.fr/tel-01746120

Houdé, O. (2018b). *L'école du cerveau*. Margada.

Houdé, O. (2018a). *Le raisonnement*. Presses Universitaires de France.

Houdé, O., Borst, G., André, C., Berthoz, A., Changeux, J.-P., Damasion, A., Damasio, H., Eustache, F., Fayol, M., Lachaux, J.-P., Peigneux, P., Prado, J., Proust, J., Jeanne, S.-F., & Johannes, Z. (2018). *Le cerveau et les apprentissages*. Nathan.

Kadosch, M. (2019). *L'analogie, modèle de l'abduction*. http://www.marcel-kadosch.eu/lanalogie-modele-de-labduction/

Kerzil, J. (2009). In *L'ABC de la VAE*. Erès. https://www.cairn.info/l-abc-de-la-vae-9782749211091-page-112.htm

Khanfir Kallel, I. (2019). Mécanismes de raisonnement possibiliste pour l'aide à la décision et l'interprétation de scènes. *Traitement du signal et de l'image*. https://hal.archives-ouvertes.fr/tel-02868499

Kleinginna, P., & Kleinginna, A. (1981). A categorized list of emotion definitions with suggestions for a consensual definition. *Motiv Emotion, 5*.

Lachaux, J.-P. (2018). Éduquer la métacognition, la clé du succès pour les enfants !
In Cerveau et psycho.

Lee, V. (2018). *A Short History of the Learning Sciences*. In R.E. West, *Foundations of Learning and Instructional Design Technology: The Past, Present, and Future of Learning and Instructional Design Technology*. EdTech Books.
https://edtechbooks.org/lidtfoundations

Le Gall, D., Besnard, J., Havet, V., Pinon, K., & Allain, P. (2009). Contrôle exécutif, cognition sociale, émotions et métacognition. In *Revue de neuropsychologie* (Vol. 1, pp. 24-33). https://www.cairn.info/revue-de-neuropsychologie-2009-1-page-24.htm?contenu=article

Leleu-Galland, E., Gallois, J.-B., & Létang, M. (2021). *Comment apprend le cerveau ? Nathan*.

Lemaire, P., & Didierjean, A. (2018). *Introduction à la psychologie cognitive*. deboeck supérieur.

Lieto, A. (2021). *Cognitive Design for Artificial Minds*. Wikipedia.
https://en.wikipedia.org/wiki/Cognitive_architecture

Lieto, A., Oltramari, A., & Vernon, D. (2018). The role of cognitive architectures in general artificial intelligence. *Cognitive Systems Research*, *48*, 1-3.
https://www.sciencedirect.com/science/article/abs/pii/S138904171730222X?via%3Dihub

Lieury, A. (2017). 35 grandes notions de psychologie cognitives. Dunod.

Lieury, A., & Léger, L. (2020). *Introduction à la psychologie cognitive*. Dunod.
https://www.sciencedirect.com/science/article/abs/pii/S138904171730222X?via%3Dihub

Masmoudi, S., & Naceur, A. (2010). *Du percept à la décision : Intégration de la cognition, l'émotion et la motivation*. De Boeck Supérieur. https://www.cairn.info/du-percept-a-la-decision--9782804137984.htm

Mercier, C., Alexandre, F., & Viéville, T. (2021a). Reinforcement symbolic learning. *The 30th International Conference on Artificial Neural Networks*.

Mercier, C., Roux, L., Romero, M., Alexandre, F., & Viéville, T. (2021b). Formalizing
problem-solving in computational thin-king : an ontology approach. *IEEE International Conference on Development and Learning*.

Métayer, M. (2012). *Qu'est-ce que la philosophie ? À la découverte de la rationalité.* Pearson (Erpi).

Miller, G.A. (1956). The magical number seven, plus or minus two : Some limits on our capacity for processing information. *Psychological Review*.

Morgagni, S. (2011). Repenser la notion d'affordance dans ses dynamiques sémiotiques. In *Intellectica*. Revue de l'Association pour la Recherche Cognitive.
https://www.persee.fr/doc/intel_0769-4113_2011_num_55_1_1170

Morin, E. (1999). *La tête bien faite. Repenser la réforme. Réformer la pensée*. Paris, Seuil.

Morin, E. (2011). *La voie. Pour l'avenir de l'humanité*. Paris, Fayard.

Nguyen-Xuan, A. (2021). *Les mécanismes cognitifs de l'apprentissage*. iSTE éditions.

Noël, M.-P. (2007). In *Bilan neuropsychologique de l'enfant*. Mardaga. https://www.cairn.info/bilan-neuropsychologique-de-l-enfant--9782870099643.htm
Norman, D. (1988). *The Psychology of Everyday Things*. Basic Books.

Oudeyer, P.-Y., Gottlieb, J., & Lopes, M. (2016). Intrinsic motivation, curiosity and learning : theory and applications in educational technologies. *Progress in brain research*.
https://hal.inria.fr/hal-01404278/file/oudeyerGottliebLopesPBR16Preprint.pdf

Pailler, J.-J. (2004). Spinoza avait raison. joie et tristesse, le cerveau des émotions, d'antonio r. damasio. In *Revue française de psychosomatique*. https://www.cairn.info/revue-francaise-de-psychosomatique-2004-1-page-165.htm?
contenu=article

Purves, D., Augustine, G.J., Fitzpatrick, D., Hall, W. C., LaMantia, A.-S., & White, L. E. (2015). *Neurosciences. de boeck supérieu*.

Rao, M. G. (1995). Bdi-agents : From theory to practice. *Proceedings of the First International Conference on Multiagent Systems (ICMAS'95).*
Reeve, J. (2017). *Psychologie de la motivation et des émotions*. De Boeck supérieur.

Romero, M. (2004). *Métacognition dans les eiah*. LIUM. Le Mans.

https://www.researchgate.net/publication/237012128_Metacognition_dansLesEIAH

Romero, M., David, D., & Lille, B. (2018). Creacube, a playful activity with modular robotics. *Games and Learning Alliance*.

https://www.researchgate.net/publication/329040986_CreaCube_a_playful_activity_withModularRobotics

Romero, M., & Giraudon, G. (2020). Le numérique va révolutionner l'éducation . . . vraiment ? https://hal.inria.fr/hal-02895694

Rougier, N. (2015). *L'intelligence artificielle, mythes et réalités*. https://interstices.info/lintelligence-artificielle-mythes-et-realites/

Roux, L., Romero, M., Alexandre, F., Viéville, T., & Mercier, C. (2020). Développement d'une ontologie pour l'analyse d'observables de l'apprenant dans le contexte d'une tâche avec des robots modulaire. https://hal.inria.fr/hal-03013685v2

Stordeur, J. (2014). *Comprendre, apprendre, mémoriser : les neurosciences au service de la pédagogie*. de boeck éducation.

Szczepanski, N. (2012). Méthodes efficaces de raisonnement en logique modale. entre de Recherche en Informatique de Lens, Université d'Artois. http://www.cril.univ-artois.fr/~szczepanski/res/Methodes%20efficaces%20de%20raisonnement%20en%20logique%20modale%20-%20Szczepanski%20Nicolas.pdf

Tardif, E., & Doudin, P.-A. (2016). *Neurosciences et cognition : Perspectives pour les sciences de l'éducation*. de boeck supérieur.

Toscani, P., Eustache, F., Devière, F., Allard, B., Asseline, S., Aubin, C., Avrillon, C., Avrillon, D., Beauperin, C., Bernard, M.F., Bourdin, F., Bourget, C., Campedel, M., David, B., Delanoue, J., Garnier, A., Gauvrit, A., Gouyette, J., Helsens, O., ... Zendrera, N. (2017). *Les neurosciences de l'éducation, de la théorie à la pratique en classe*. Chronique Sociale. Auteur(e)s membres du GRENE :Groupe de Recherche En Neurosciences Educatives.

Vacher, Y. (2011). La pratique réflexive. *Recherche et formation*, 66.
<figure><img src="image_25.png" /><figcaption>Figure 23 - Selection of an activity in the activity space</figcaption></figure>

This figure is inspired by a diagram by Benjamin Clément [Clément, 2018]. It represents an example of activity
selection in the activity space (gray shape) with four hierarchical groups (orange shapes). Each group has one or more parameters (yellow rectangles) which themselves have values (green shapes). The thin blue arrows show the hierarchy between the groups while the big blue arrow represents the selection process pointing to a big blue rectangle. This one is a selection possibility (an example of activity selection). The small blue rectangles on the left and right represent the possible combinations of values at the end of each group.

<figure><img src="image_26.png" /><figcaption>Figure 24 - Zone of proximal development and Flow.</figcaption></figure>

This figure is a diagram made by Benjamin Clément (Clément, 2018). It is a graph representing the states in which a learner may be when performing an activityª. The level of difficulty is on the y-axis while the learner's skill level is represented on the x-axis.

ª"Basawapatna et al. (2013) propose the concept of Zones of Proximal Flow where they come up with the idea of the Zone of Proximal Development being located in between regions of Flow and anxiety" (Clément, 2018).

<figure><img src="image_27.png" /><figcaption>Figure 25 - Example of an activity graph with evolution of the zone of proximal development.</figcaption></figure>

This figure is a diagram done by Benjamin Clément (Clément et al., 2015). It is an example of a breakdown of activities made by an educational expert. The activities are organized by level of difficulty and type. The zone of proximal development (ZPD) represents all the activities de l'espace d'activité pouvant être proposées pour favoriser le plus efficacement possible le progrès d'apprentissage.