Developing Competence in Interactive Systems

Bénédicte Talon, Mouldi Sagar, Christophe Kolski

► To cite this version:

Bénédicte Talon, Mouldi Sagar, Christophe Kolski. Developing Competence in Interactive Systems: The GRASP Tool for the Design or Redesign of Pedagogical ICT Devices. ACM Transactions on Computing Education, 2012, 12 (3), pp.1-43. 10.1145/2275597.2275598 . hal-03285232

HAL Id: hal-03285232

https://uphf.hal.science/hal-03285232

Submitted on 21 Dec 2023

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
A version of this pas has been published in:
ACM *Transactions on Computing Education*, volume 12, issue 3, pp. 1-43, 2012.

The final version is available in the ACM Digital Library: http://dl.acm.org/

# Developing Competence in Interactive Systems: The GRASP tool for the design or redesign of pedagogical ICT devices

BENEDICTE TALON, Univ Lille Nord De France

MOULDI SAGAR, Univ Lille Nord De France

CHRISTOPHE KOLSKI, Univ Lille Nord De France

Training professionals to design and produce interactive systems requires the use of well thought-out training scenarios. Indeed, it is essential to consider pedagogical forms whose objective is providing learners with field experience. It is also necessary to motivate learners to learn about subjects that sometimes seem distant from their immediate concerns and to renew their interest in these subjects. This paper presents a tool, called GRASP, that helps design pedagogical devices. This tool was first evaluated through three educational initiatives implemented in two universities in the north of France. These initiatives were analyzed to enhance the tool. Then, a second evaluation was conducted to validate the tool more widely. The first part of this paper presents the scientific context. Then are introduced the design tool, the evaluation protocol, and the three devices built using this tool and their evaluation. Some of the lessons learned during the initiatives are described. Next, the second evaluation, conducted with more teachers, is presented. Finally, the conclusion highlights the perspectives opened by these initiatives.

Author's addresses: B. Talon, Univ Lille Nord de France, F-59000 Lille / LISIC, ULCO, F-62228 Calais, France. M. Sagar, Univ Lille Nord de France / DeVisu, UVHC, F-59313 Valenciennes, France.C. Kolski, Univ Lille Nord de France / LAMIH, UVHC, F-59313 Valenciennes, France / CNRS, UMR 8201, F-59313 Valenciennes, France.
# 1. INTRODUCTION

This publication focuses on engineering education, particularly on the way to develop skills for interactive systems development in a professional environment. In this field, the skills to acquire are numerous and varied [Gibson, 2005]; these skills can also be also multidisciplinary especially if user-centered approaches are targeted [Hewett et al., 1996; Jacko and Sears, 2003].

Transmissive education is still widespread in the French university system. It is essentially based on lectures followed by sessions in which the theory of the lecture is applied through exercises and restricted cases. Years of experience with this kind of pedagogy have shown us its limits. Although the learners do manage to acquire the fundamental knowledge of the domain, they are frequently aware of the importance of mastering diverse know-how only late in the game, and their capacity to transform their knowledge into competence is often only developed during their internships, or even during the first job in their chosen profession. Very often, they require additional training in the company they end up working for.

Our teaching experience has shown that professional skills can successfully be taught in interactive systems engineering. Montmollin (1986) affirms that the achievements of professional history allow anticipating phenomena, understanding the implicit instructions and expecting the variability of the tasks. It is thus important to provoke situations that will allow the construction of the work history of every future engineer.

Given this French context and our commitment to professional training, pedagogical devices<fnref n="1" /> have been developed within our university context. They better support and facilitate learner professionalization and thus better respond to company requirements. These devices allow the fundamental professional know-how to be acquired, as well as a capacity to use this know-how when faced with a complex situation.

Educators like Dewey defended another form of pedagogy [Dewey, 1919] in which it was essential to end the juxtaposition of theoretical education and practical training. Nevertheless, our French educational context is often filled with inevitable constraints: high numbers of learners, few material resources, few hours for each teaching unit, imposed classroom setups and obligatory academic rhythms.

In this article, some of the important concepts related to professionalization are defined and connections between these concepts are established. These concepts are then integrated into an initial framework, which forms the basis of our proposition. Forms of pedagogy able to develop professional competences are then discussed. A status report of professional education programs is also provided, with a special attention toward the field of interactive systems. A manageable tool, called GRASP (GRid-bASed Pedagogical design), for helping the design or redesign of pedagogical devices is then introduced. GRASP asks a number of questions about the intended learning. This tool was first evaluated by developing three pedagogical devices. An evaluation of each device allowed to test the feasibility of using this tool in a university environment. A second evaluation conducted with a larger population allowed us to validate its usability and to obtain promising results. The article ends with an analysis of our work and a presentation of the perspectives opened.

<footnote n="1">¹ A pedagogical device is defined as a set of means (methods, tools, procedures, principles of action, stakeholders) to support the learning process as required by the needs of the learner [Talon and Leclet, 2008]. The device designed for a teaching unit contains all constituents leading to the learning situation (e.g., organization of teaching space, organization of time, technological means given to learners, assessment).</footnote>
## 2. INITIAL FRAMEWORK

A review of the literature has been conducted in several fields: professionalization, pedagogy and professional training in engineering, particularly in interactive systems development. Since our aim is to train Interactive System professionals who will be competent in their professional context and will be able to develop expertise later, it was necessary for us to identify an educational frame of reference—in fact, a reference unit—in order to define training objectives and to provide a basis for comparison. Some of the results of the investigations are reported in this section. Figure 1 summarizes the concepts that are of interest in this article.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Our representation of the concepts of competence from the literature review</figcaption></figure>

### 2.1 Professionalization: competence, knowledge, know-how and expertise

In 2001, Weinert described competence as

> "a roughly specialized system of abilities, proficiencies, or skills<fnref n="2" /> that
> are necessary to reach a specific goal".

According to Basselier,

<footnote n="2">² Skills are observable when an "activity reached an elevated level of interiorization, which particularly occurs with the prompt and precise delivery, as well as the capacity to act as parallel of other activities" [Leplat and Pailhous, 1981].</footnote>
"competence is thus non-routine and embodies the ability to cope with complex changing environments" [Basselier, 2001].

Professional competence covers all that makes it possible to react appropriately in a given context: knowledge, aptitudes and attitudes. According to the French AFNOR standard X50-750, competence is the

"implementation, in a professional situation, of capacities that make it possible to properly hold a position or exercise an activity" [AFNOR, 1992].

The AFNOR definition was chosen as appropriate to the context of professional training.

In addition to competence, professional qualifications have an important place. The Montmollin's definition of professional qualifications is:

"a set of professional and social knowledge and know-how that allows workers to control their work, while being within the company" [Montmollin, 1986].

The following concepts are related to competence:

**Knowledge** was defined by Eraut (2000) as

"what people bring to practical situations that enables them to think and perform. Such personal knowledge is acquired not only through the use of public knowledge but is also constructed from personal experience and reflection."

**Know-how** is made up of the steps, methods and instruments whose practical application a person has mastered [Le Boterf, 2002]. There are three types of know-how: operational know-how<fnref n="3" />, cognitive know-how<fnref n="4" /> and social know-how<fnref n="5" />.

**Expertise** is the result of developing knowledge about field and the way of solving problems, as well as all the metacognitive capacities related to work. According to the literature, approximately 10 years of work in the field of expertise, with the desire to improve, are needed to become an expert. During these years, professionals acquire experience by dealing with various situations in which they learn how to behave, thus developing knowledge about their field and the way of solving problems, as well as all the metacognitive capacities [Upchurch and Sims-Knight, 1999]. Expertise takes time to develop, requires effort and is difficult to transmit directly. An experiment conducted by two of the authors of this article showed the correlation between the level of expertise and the capacity to detect defects in multimedia applications [Huart, Kolski and Sagar, 2004], notably proving that experts develop a capacity to detect defects related to their field of expertise more easily than those related to other fields.

Thus, for example, a first year undergraduate learner in a short cycle in computer science must develop a competence in standardization, which is necessary to design relational databases. This competence is essential to the database administration qualification. Standardization produces a set of relations composed of the properties necessary for managing a business. This process is necessary to prevent possible database corruption (e.g., insertion, updating and deletion of anomalies). As part of the curriculum, the teacher first presents the knowledge of the normal forms ("know

<footnote n="3">³ Skill set required to execute, reproduce and adapt procedures required by a job.</footnote>

<footnote n="4">⁴ Skill set required to acquire knowledge in order to analyze and understand the world, exchange and communicate with others and solve problems.</footnote>

<footnote n="5">⁵ Skill set required to establish reciprocal relationships with positive partners (e.g., giving and taking fairly, developing conditional cooperation) in disciplines implicit in daily life, while implementing such success factors as motivation, anticipation, positive self-image, sense of responsibility and control of space.</footnote>
the normal forms”). Based on this **knowledge**, he/she then develops the **operational know-how** to “standardize a database in 3NF”, which depends on the **cognitive know-how** to “manipulate a dependence graph” introduced in the mathematics module. Subsequently, through group work, he/she develops the learners’ **social know-how** to “communicate and negotiate”, which is a central concern.

## 2.2 Pedagogical theories and models - Moving the learner towards competence and professional qualifications

Promoting the development of appropriate pedagogical environments is a concern. These environments must favor the learning of identified professional skills. Shuell (1986) considers that a student has learned when an **enduring change** can be measured in his/her behavior or in his/her capacity to behave in a given fashion.

Learning theories consider the variables that influence the learning process and provide explanations on how this influence occurs. They propose answers to critical issues, such as “How does learning occur?”, “What is the role of motivation?”, and “How does transfer of learning occur?”. By giving answers to these fundamental questions, they provide a framework, which serves as the bridge between research and education [Schunk, 2012]. Learning theories help to define educational frameworks, which define the principles through which the theory can be applied to learning and teaching practices.

In the literature, three major theoretical supports to the instructional models are usually identified [Schunk, 2012; Ally, 2008; Mayes and De Freitas, 2004]. The first is a historical trend of behaviorism; the others are the more recent cognitivist and constructivist models. Each of these theories makes different assumptions about what is crucial for understanding and has an influence on the modalities for implementing an instructional model.

### 2.2.1 The behaviorist perspective

Behaviorism [Thorndike, 1913; Watson, 1913; Skinner, 1974] postulates that learning is an observable change in behavior caused by external stimuli in the environment. It is the observable behavior that indicates whether or not the learner has learned something, and not what is going on in the learner's head, which is considered as a black box. Explanations for learning don't include observation of internal events, not because they don't exist, but rather because the causes of learning are observable events. According to Skinner (1953), a response to a stimulus is more likely to occur again in terms of the consequences of prior response: reinforcing consequences make the response more likely to occur, whereas punishing consequences make it less likely.

According to the behaviorist model, competence is built step by step through sequences of activities. Learning is the forming, strengthening and adjusting of the knowledge and know-how, particularly the reinforcement via feedback. Each student responds to questions or problems and receives immediate feedback on his/her response.

Corresponding pedagogical models are prescriptive models. In these models, teachers control most classroom events and present very structured lessons with a step-by-step progression between subtopics. They use many examples and demonstrations. The teacher defines sub-goals and implements progressive exercises for the various stages of difficulty. The role of the student is to practice the exercises by following the marked route. The student's errors indicate sub-objectives poorly or insufficiently decomposed. Therefore, in the case of engineering learning, the teacher needs to concentrate on the key concepts of the domain, the main goals and sub-
goals, and the current level of student skills. He/she determines the necessary steps to develop a correct behavior.

Pedagogical models respecting behaviorist theories have been designed, such as Programmed instruction [Skinner, 1958], Mastery Learning [Bloom, 1968], Direct instruction [Engelmann, 1997], and the QAIT model [Slavin, 1997].

### 2.2.2 The cognitivist perspective

For behaviorists, the observable behavior indicates whether or not the learner has learned, and not what is going on in the learner's head, which is considered as a black box. This vision has evolved in the interim. Now, some psychologists claimed that not all learning is observable and that learning is more than a change in behavior. As a result, in the 1970s/1980s, there was a shift away from behaviorist to cognitive learning theories where theorists [Gagné and Briggs, 1974; Ausubel, 1974; Bruner, 1980] emphasize understanding and thinking ability. In the cognitive approach, learning is seen as an internal process. The development depends on the processing capacity of the learner, the amount of effort he/she expends during the learning process, the depth of the processing, and the learner's existing knowledge structure [Craik and Lockhart, 1972; Ausubel, 1974]. It is consequently important to know processes that will promote acquiring competence.

Andre (1986) listed several suggestions that are useful for training students, such as providing students with metaphorical representations or having students verbalize during problem-solving (verbalization of thoughts can facilitate problem solutions and learning), using questions that require them to practice concepts they have learned, providing examples, and teaching strategies that help learning and problem-solving. The literature gives priority to student-centered methods, such as discovery learning [Bruner, 1961], nine events instruction [Gagné and Briggs, 1974], and cognitive apprenticeship [Collins and al., 1988].

### 2.2.3 The constructivist perspective

For constructivists like Dewey (1960), Vygotsky (1978) and Piaget (1980), learners learn by observation, processing, and interpretation, and then personalize the information into personal knowledge. Van Merrienboer and Kirschner (2001) claim that “the what-to-teach question rests on a description of real-life or professional tasks”. For constructivist theorists, learners interpret information and the world according to their personal reality. Learners learn best when they can contextualize what they learn for immediate application to acquire personal meaning. In a behavioral model, there is only one correct view of the learning object; in a constructivist model, there are as many learning objects as students [Dessus, 2008].

Understanding is gained through an active process of creating hypotheses and building new forms of understanding through activity. The assumption is that conceptual development occurs through intellectual activity rather than by the absorption of information through instruction. In a constructivist model [Dessus, 2008], the instructional process is nonlinear and sometimes chaotic; planning is developmental, reflective and collaborative. The work objectives appear from the design and development. There are no experts regardless of the content taught, and the educational purpose is learning in meaningful contexts. The formative assessment is the most important, and the subjective data may be the most significant.

In these pedagogical models, the teacher's role is complex. He/she must first identify recurring obstacles and set up situations designed to raise the student's awareness of the inadequacy of his/her views. Finally, he/she should help students to build new knowledge and to consolidate them by ad hoc exercises. The student's role is to grasp the meaning of the problem, invest his/her initial knowledge, accept the destabilization of the process, and recognize the need to progress. Finally, with the
help of the teacher, he/she must build new knowledge and consolidate it with ad hoc exercises.

Literature gives priority to student-centered methods. This covers several forms of pedagogy, such as

— Project-based pedagogy, in which

> "Students gain a deeper appreciation of theory when it is grounded in practice, and the practical environment gives rise to opportunities to improve professional skills" [Gibson, 2001];

— Pedagogy based on problem solving, which makes it possible for students to learn while solving a problem [Akınoğlu and Tandoğan, 2007];

— Pedagogy based on simulation or role-playing, in which students are placed in a virtual environment that simulates a professional context and must act accordingly. The interested reader can consult Sims-Knight, Upchurch and Fortier (2005) and Baker, Navarro and Van Der Hoek (2005) for several examples of simulation environments.

Constructivism is often placed in a situated perspective known as socio-constructivism. Learning should not be viewed simply as the transmission of abstract and decontextualized knowledge from one individual to another, but a social process whereby knowledge is co-constructed (community of practice). This perspective suggests that learning is situated in a specific context and embedded within a particular social and physical environment. Goodyear (2002) gives an account of networked learning as knowledge sharing for continuous professional development. He describes a cycle of learning, moving through phases of externalization (of tacit knowledge), sharing, discussion, refinement and then internalization. According to him, the design of the learning tasks is central.

### 2.2.4 Discussion.

We have reviewed various theories of learning and associated teaching models. The question is now what kind of environment does a teacher design to engage students in the appropriate activities in order to develop targeted knowledge and skills. Teachers can use an environment that blends the elements that emphasize all three perspectives: learning as behaviour, learning as the construction of knowledge and meaning, and learning as practice.

According to Ertmer and Newby (1993) and Mayes and De Freitas (2004), the schools of thought can be used as a taxonomy for learning. For example, behaviorist strategies can be used to learn the “what” (i.e., facts), cognitive strategies to learn the “how” (i.e., processes and principles), and constructivist strategies to learn the “why” (i.e., higher level thinking that promotes personal meaning and contextual learning). Consequently, whatever the referenced theory is, the teacher must have a clear idea of what students must be able to know and do at the end of a teaching unit. In our particular case, students must be prepared to be competent engineers, making pertinent decisions in their future careers. It is essential that the outcomes of teaching units mirror this.

We agree with Biggs (1999) who claims that a good pedagogical design is one that insures that there are absolutely no inconsistencies between the curriculum, the teaching methods, the learning environment, and the assessment procedures. What students must know or be able to do at the end of the session corresponds to what Biggs calls Intended Learning Outcomes (ILOs) in his constructive alignment theory [Biggs, 1999]. According to him, it is important for a teacher to communicate the Intended Learning Outcomes to students so they can share in the responsibility of achieving them. Students will then tend to look at the assessment and will structure their learning activities in order to optimize their performance on the assessment.
The teacher must therefore make sure that the assessment does indeed test the learning outcomes he/she wants students to achieve. The ILOs, the learning activities and the assessment must all be aligned.

To this end, a teacher must define ILOs he/she aims to develop, design activities that will produce the development of these outcomes, and define criteria that will engage the students to respect these activities, thus guiding them towards real skills acquisition. A correct assessment is criteria-based where grades are awarded according to how well students meet the desired learning outcomes. According to Biggs, to promote deep learning, approaches must be considered that require more active participation. Deep learning involves the critical analysis of new ideas, linking them to already known concepts and principles, and leads to understanding and long-term retention of these concepts so that they can be used for problem solving in unfamiliar contexts. Therefore, teachers should consider activities that encourage that behavior, as suggested in our exploration of learning theories above.

To encourage a deep learning approach, the pedagogy should move from a system centered on information focused on critical thought, dialogue, discussion and deliberation. Thus, the training situation should suggest questions to students, who, by finding the responses, acquire additional experience and new skills. Students construct their knowledge base through researching situations that force them to carry out a task, produce something or solve a problem. As Basselier affirms,

> "Managers build their ability to develop relevant knowledge and understanding of IT over time through their participation in IT initiatives. [...] Experience is thus the basis for the development of tacit knowledge" [Basselier et al., 2001].

Teachers guide and facilitate the tasks, by organizing the training situation and making the essential resources available. Barr and Tagg (1995) have provided a complete presentation of this paradigm shift, from teaching, or instruction, to learning

## 2.3 Professional education in engineering and interactive Systems

The professionals who create the products must be equipped with the skills that will allow them to elaborate products that meet their customers' needs.

However, in general, the learners taught by the various university engineering programs are lacking such competences as project management, teamwork, and/or oral and written communication [Wanous, Procter and Blamey, 2006]. This is mostly because such programs do not bother to teach the job skills that are judged crucial by employers. In fact, though theoretical knowledge provided by such organizations is often excellent, they are frequently reproached for training engineers who lack practical knowledge and experience.

In this larger context of engineering, Interactive Systems engineering works to define, design and evaluate the methods, components and systems in interactive systems. Developing and producing software is difficult, and this relatively young discipline is confronted with specific constraints that other disciplines do not encounter: software "invisibility", deceptive flexibility, needs that are difficult to identify and to model, to name only a few. Customers complain about the lack of product quality and the companies denounce the lack of qualified personnel [Callahan and Pedigo, 2002; Conn, 2002]. Project success rates are still relatively low, with 32% of successful projects in 2009 [Standish Group, 2009], though these results raise some questions about their collection and analysis modalities [Glass, 2006; Eveleens and Verhoef, 2010]. The software profession is making great strides toward increasing the success rates [Nidiffer, 2007].

But, in many countries, universities can be reproached for not producing the learners who have skills, knowledge and attitudes to satisfy the needs of the data-
processing job market. Lacks of skills in communication and project management, which are considered essential by recruiters, are frequently cited. The results of a survey carried out by Lethbridge (2000) show that the knowledge developed at the university and the kind of knowledge required of computer science specialists on the job doesn't match. The participants in this study revealed that, when leaving the educational system, learners lack the skills that are considered essential, such as project management, written expression, human factors and organization consideration, for example.

Some of the disciplines highlighted by the surveys are of interest in this paper. It is essential to adopt teaching practices that will inspire the understanding needed to motivate learners, while emphasizing the precision of the processes taught. This means not only reflecting on appropriate program content but also changing and adapting teaching practices to include a more active pedagogy.

### 3. PROPOSING A TOOL THAT HELPS TO DESIGN OR REDESIGN PEDAGOGICAL DEVICES: GRASP

In order to develop a pedagogical device that fully meets their expectations, teachers must consider the specific elements of their context. In agreement with the ideas defended by Biggs, we think that teachers have to identify their intended learning outcomes precisely and promote approaches that encourage a more high-level learning. To design and build a device, teachers may make use of an existing instructional design method [Paquette et al., 1997]. Instructional Design is an approach to develop effective learning materials for different learning contexts through a defined process.

Different models of instructional design have been proposed. Among them the often-cited ADDIE (Analyze, Design, Develop, Implement and Evaluate) model<fnref n="6" />, the Nine Events of Instruction [Gagné, 1985] or the Systems Approach Model [Dick and Carey, 2005] can be found. The problem with these models is that though they meet the needs in terms of devices, they are often cumbersome to implement and require that the individual teacher master the model's techniques.

In a study led by Wedman and Tessmer (1993), they observed that designers reported using instructional design activities of the suggested model but omitted one or more activities on every project. Among reasons for excluding an activity were "not enough time" and "considered unnecessary". In fact a study done by Kenny and al. (2005) shows that teachers don't follow the prescriptive process of instructional design methods: "But in practice, we suggest that models of instructional design, while implicitly prescriptive, are in fact conceptual frameworks for practice. ID models are useful to designers and inform practice, but few if any designers actually use models to confine their practice". Other models have been suggested to limit the time of development such as the rapid prototyping [Tripp and Bickelmeyer, 1990], which is an alternative design strategy based on the design, development and review cycle.

The approach proposed in this paper is different. The aim is rather to help teachers build pedagogical devices by getting them to ask pertinent questions. To accomplish this, they will have to use a tool in the form of tables (see Tables I, II and III). This tool is called GRASP (GRid-bASed Pedagogical design). Each cell in the table will lead the teacher to ask questions, and their responses are summarized in this table. The GRASP tool responds to a desire to simplify the process. It is based on our experience and includes elements that direct the teacher in a rapid design process. The first commitment is to meet the expectations of teachers in their desire

<footnote n="6">⁶ The ADDIE model is a colloquial term used to describe a systematic approach to instructional development [Molenda, 2007].</footnote>
to design or redesign the educative methodology. GRASP is based on the observation that the teachers are willing to change their systems but do not want to perform too long of a process. This confirms the observation of Wedman and Tessmer (1993). The second evaluation (described below) showed teachers in higher education don't have much disponibility. Teachers have multiple tasks to perform: teaching, research and administration. They want to perform all these tasks professionally but do not have the time to perform too long of a design process.

Based on this observation, we developed GRASP, which incorporates the following elements:

- **A process guided by the ILOs:** GRASP was developed so that, when designing a device, teachers consider the competence components described in Figure 1. This allows them to focus on the intended learning objectives before designing the device, according to Biggs' recommendation to build a device keeping in mind the intended learning outcomes. In GRASP, the ILOs are described in terms of professional skills to be attained. This description concerns the first part of GRASP, where teachers explain all the elements in the competence description section: knowledge, operational know-how, social know-how and cognitive know-how. Filling in this section allows teachers to focus on outcomes.

- **A systemic vision:** Our Software Engineering background made us familiar with approaches in which the constraints are met by solution design. We defend systemic approaches in which the system is seen as a set of elements in dynamic interaction, organized in terms of a goal. Systemic approaches provide a general method for solving complex problems. They have been applied in various fields, such as the product design, economics, and education. They are a good possibility for designing a pedagogical system [Chen and Stroup, 1993]. The systemic approach is defined in general in five stages: problem definition, problem analysis, solution plan development, solution plan evaluation and solution plan revision. The goal is to break down the problem into sub-problems that are easier to deal with. This process can be described as heuristic because it suggests the means but does not guarantee the result. In GRASP, the first three steps break down the tool into 3 parts. Part 1 defines the problem by identifying the outcomes (i.e., skills). Part 2 analyzes the problem, identifying the context (i.e., public, strengths and constraints). Part 3 elaborates the solution to the problem by determining the role of the teacher and the student and the presence/distance distribution. The process is described in a scenario for the development of the activities needed to reach the ILO. The resources necessary to carry out the activities must be anticipated in order for the teacher to be prepared. ICT are to be envisaged to better support the future scenario.

- **Iterative assessment:** There is no obvious right or wrong solution. The assessment must be ongoing in order to move toward the solution by successive approximations. Implementation and evaluation correspond to stages 4 and 5 of the systemic approach. Using an evaluation process model, such as the one proposed by Pears and Daniels (2003), will be helpful in evaluating the device.

- **A preference for active approaches:** Although GRASP does not close the door to some other relevant educational activities, GRASP develops active approaches, which are better able to promote skills acquisition.

GRASP was designed according to these perspectives. The pedagogical device is developed through steps. Each step gathers information, and the idea of the pedagogical device is refined through these steps.

The first part of GRASP (Table I) focuses on target skills (the Intended Learning Outcomes). Teachers should first focus on their educational objectives: what are their
intended learning objectives in terms of knowledge and know-how. At this point, they should be careful to limit their demands. This preliminary work helps to identify the module's educational requirements.

<table><caption>Table I: GRASP – Targeted skills</caption><thead><tr><th colspan="2">Method: Give a name which summarizes your general method</th></tr></thead><tbody><tr><td>Skill targeted by the module</td><td>Summarize in one sentence the skill covered by the device<br>It must be clear in your mind. Avoid having too many important requirements.</td></tr><tr><td>Knowledge</td><td>Detail knowledge required for this skill – The Know What. List what will allow the learner to think and perform.<br>You must separate the knowledge the learners have prior to the module from the knowledge that will be acquired during the module.<br>You must separate the theoretical knowledge (know what) and the procedural knowledge (know how).<br>You must answer the following questions: "What knowledge has already been acquired (theory and procedure)" and "What knowledge do the learners need to acquire (theory and procedure)?"<br>Do not forget to cover domain knowledge and transverse knowledge.</td></tr><tr><td>Operational Know-How</td><td>Describe the different procedures (know how to operate) learners must master to be able to act. List the skill set required to execute, reproduce and adapt procedures required by a job.<br>You must separate the procedures the learners have already mastered prior to the module from the procedures that will be acquired during the module.<br>You must answer the following questions "What must a learner be able to do and to reproduce?" and "What will a learner be able to do and to reproduce after the module?"</td></tr><tr><td>Cognitive Know-How</td><td>Describe the different capacities that learners must master to function: capacities to process information, reason, learn, and solve problems. List the skill set required to acquire knowledge in order to analyze and understand the world, exchange and communicate with others and solve problems<br>You must answer to the question: "What must a learner be able to do?" and "What will a learner be able to do after the module?"</td></tr><tr><td>Social Know-How</td><td>Describe the different social capacities learners must master to act in the targeted domain. List the Skill set required to establish reciprocal relationships with positive partners (e.g., giving and taking fairly, developing conditional cooperation) in disciplines implicit in daily life, while implementing such success factors as motivation, anticipation, positive self-image, sense of responsibility and control of space.<br>You must answer the following questions: "What social qualities are expected for learner to be able to act?" and "What social qualities are to be developed by the module?"</td></tr></tbody></table>

The second part of GRASP (Table II) gathers the constraints under which the device will operate. Defining the educational context allows teachers to better understand their context, analyze the constraints and guide the choices for the device.
<table><caption>Table II: GRASP – Context</caption><thead><tr><th></th><th></th></tr></thead><tbody><tr><td>Public Targeted</td><td>Define your public:<ul><li>Level</li><li>Total number of learners</li><li>Number of divisions of the class</li><li>Specific difficulties: list the specific difficulties of your public (e.g., agitated, decentralized, etc.)</li><li>Specific advantages: list the specific advantages of your public (e.g., of good will, handles the computer well, etc.)</li></ul></td></tr><tr><td>Module Period</td><td>Define the possible duration and their constraints<ul><li>Session length</li><li>Session frequency</li><li>Possibility of varying the length or frequency?</li><li>Homework?</li></ul></td></tr><tr><td>Constraints</td><td>Identify the different constraints under which you work.<br/>List the specific difficulties (e.g., technical, organizational)</td></tr><tr><td>Strengths</td><td>Identify the different strengths of your learning environment<br/>List the specific advantages of your teaching environment (e.g., human resources, pedagogical resources, equipment)</td></tr></tbody></table>

The third part of GRASP (Table III) facilitates the design choices for the pedagogical device: role of the teacher, available resources for the learners, type of process, etc. The choices depend on the teaching objectives and the specific context of the module. By addressing different issues, the teachers define their teaching scenario. The interested reader will find useful information on making choices and completing the table in an article written by Debbie Richards [Richards, 2009]. In this article, the author considers key design choices of project-based courses.

Information and Communication Technology (ICT) can be a good mediator for communication, document management or publishing. Studies show that remote learners can outperform traditional classroom students [Zang et al., 2004]. The device's ICT instrumentation can be easily obtained by making use of Web 2.0 tools (e.g., blogs, chats, forums and wiki) or by deploying activities on an available Learning Management System.
<table><caption>Table III: Tool for designing a teaching device – Organizational choices</caption><thead><tr><th></th><th></th></tr></thead><tbody><tr><td>Teacher Role</td><td>Define the role of the teacher. Choose among:<ul><li>Strengths: continuous monitoring of the progress of activities; anticipation of questions and problems.</li><li>Participatory: an active presence among the learners, facilitates frequent and regular monitoring; anticipates questions and problems.</li><li>Intermediate: Monitoring the activity without interfering in the learners' work.</li><li>Weaknesses: provides assistance for learner work; launches and recovers the deliverables at the end of activity.</li></ul></td></tr><tr><td>Type of pedagogy</td><td>Select and justify the choice of pedagogy: type and organization. It is advisable to encourage forms of active learning in groups that promote social skills learning and mobilize of diverse competences: project based pedagogy, pedagogy based on problem-solving, simulation, games, for example</td></tr><tr><td>Resources Given</td><td>Identify the resources (and their format) you have to provide to the learners:<ul><li>Course materials,</li><li>Advice for carrying out the activities</li><li>Scenario,</li><li>Methods,</li><li>Tools,</li><li>Document models,</li><li>Evaluation criteria,</li><li>Etc.</li></ul></td></tr><tr><td>Kind of Process (free, guided...)</td><td>Define and justify the type of educational process that you want the learners to implement, in line with the teacher role:<ul><li>Completely guided,</li><li>Completely free (only the outcome is important),</li><li>Intermediate process (timepoint for product delivery).</li></ul></td></tr><tr><td>Evaluation mode</td><td>Define precisely your evaluation modes:<ul><li>Result evaluation,</li><li>Process evaluation,</li><li>Collaborative work evaluation,</li><li>Documentation quality evaluation,</li><li>Etc.</li></ul>Do not forget to connect your evaluation to the skills targeted by the module. The learner must be motivated by the evaluation.</td></tr><tr><td>Working method</td><td>Define the learner and teacher activities precisely. A schedule is necessary if you have chosen a completely guided process or an intermediate process. You can produce a guideline that will support your sessions and will be distributed to learners.</td></tr><tr><td>Work Organization (In presence of the teacher/outside of the classroom /mixed)</td><td>Choose an organizational style for the work:<ul><li>In the presence of the teacher only,</li><li>Outside of the classroom only,</li><li>An hybrid work-style.</li></ul>Describe the distribution of tasks during the sessions if you choose an hybrid work</td></tr><tr><td>ICT Instrumentation</td><td>Study and propose ICT instrumented solutions that could help to deploy the device and foster learner work<br>You can choose to use collaborative platforms, such as Moodle or available technologies, such as Wiki, Blog, e-mail, chats and/or forums.<br>Remember to use a variety of tools<ul><li>Editorial</li><li>Collaboration</li><li>Communication</li><li>Development</li></ul>which can promote learner activities and facilitate your work.<br>It is important to specify the format of deliverables that you wish to obtain: pdf, doc, jpg or another format.</td></tr></tbody></table>

As a preliminary step of GRASP evaluation, the three authors used it to develop pedagogical devices in their respective universities. This evaluation resulted in
GRASP's first improvement, especially at the field level to provide information and
explanations. The kind of device was then studied, as well as their appropriateness.
The result of this improvement is presented in an appendix: Table IX. Another
evaluation was then set up to test the validity and usability of GRASP V2.0 with a
larger public. Its purpose was to detect GRASP errors and observe the difficulties
encountered by teachers while filling in the GRASP tables. The following two
sections present our first evaluation process and results.

# 4. THE EVALUATION PROCESS OF GRASP, VERSION 1

To structure this study, the model proposed by Pears and Daniels (2003), and thus
their terminology, were used.

**4.1 Study Object**

The hypothesis is that GRASP is usable. The teachers can use GRASP to help them
structure their learning objectives and context in order to develop devices directed to
their professionalization goals.

**4.2 Study Instance:**

Teachers: 2 Computer Science teachers and 1 Human-Computer Interface (HCI) ergonomist.

Learner cohort: a class of 56 second-year learners learning about database standardization, a class of 25 fifth-year learners learning about evaluating interactive systems (i.e., websites), a class of 25 fourth-year learners learning about HCI specification for complex systems.

Environment: Information Technology IUT (ULCO), Master (UVHC)

Time and place: January to June 2008 at the university, either ULCO or UVHC.

**4.3 Tools and techniques tested:**

GRASP V1 (GRASP comes in the form of tables, with the teachers filling the cells in a specific order).

**4.4 Learning and teaching theories:**

Behaviorist, Cognitivist and Constructivist theories are possible. However, learning
forms that encourage competences acquisitions are recommended.

**4.5 Educational Domain**

Interactive system competence education.

**4.6 Study approach**

Three teachers used GRASP to design or redesign a device appropriate to their needs
and constraints in their teaching units. Then, they deployed their devices in their
universities.

**4.7 Data collection**

The following data collection technique was performed:

- A teacher questionnaire about the use of the tool and the arrangements of the device in place,

- A collection of the deliverables produced during the course, and

- A survey of learner satisfaction with the device and the difficulties encountered.
## 4.8 Data analysis

A research approach based on statistical information collected from the teacher (i.e., satisfaction rate, difficulties) and the learners (i.e., satisfaction rate, difficulties) was applied. This approach was implemented by analyzing the data collected via the questionnaire and interviews. Empirical analysis involves determining if there is a perceived improvement in learner attitude or a visible satisfaction at the end of the teaching unit. The learners' acquisitions of domain competences were measured by analyzing their deliverables and/or a final assessment.

## 4.9 Study Outcome

Section 5 presents the three devices designed using our tool and the main outcomes of the first evaluation. The overall results of the evaluation are provided.

# 5. PRELIMINARY EVALUATION - DEVICES FOR DEVELOPING SKILLS IN THE DOMAIN OF INTERACTIVE SYSTEMS AT TWO FRENCH UNIVERSITIES

In a preliminary evaluation of GRASP, the three authors used it to produce a pedagogical device in the context of their teaching unit. The three case studies are described below.

## 5.1 Case 1: a collaborative project-based approach focusing on interactive system design

### 5.1.1 Context

The first author of this article (hereafter referred to as author 1) teaches in classes in the Information Technology department of the University Institute of Technology (IUT) at the University of Littoral - Côte d'Opale (ULCO). Faced with difficulties for transmitting know-how, she wanted to try a new pedagogy in database design teaching unit. The project objective was to put learners in a real situation with the need to standardize a database before it was embedded in the Oracle database management system.

### 5.1.2 Device

This teacher adopted our tool to design the Database module, intended for undergraduate learners in a short cycle class. The answers to questions are summarized in Table IV. As mentioned above, the first part of this table describes the skills targeted by the device (white background), the second part (light grey background) describes the constraints, and the third part describes the pedagogical device (grey background).

While aiming to develop the skills needed on the job (e.g., designing 3NF databases), the pedagogical device must develop and/or reinforce transversal competences in order to remedy the deficiencies in project management (e.g., planning, organization, resource management) as well as in communication (e.g., written and oral communication, negotiation).

<table><caption>Table IV: A device for developing competences in database normalization</caption><tr><td colspan="2">A collaborative project-based approach to acquire database standardization skills<br/><i>A device for the Relational Database teaching unit</i></td></tr><tr><td><b>Skill targeted by the teaching unit</b></td><td>Knowing how to design and query a 3NF database</td></tr><tr><td><b>Knowledge</b></td><td><ul><li>The concept of relation</li><li>The advantages of standardization</li><li>Normal forms: 1NF, 2NF and 3NF</li><li>Functional dependencies</li><li>Stages of a standardization process</li><li>SQL Language: DML/DDL</li></ul></td></tr></table>
<table><tr><td></td><td>To be transmitted by lecture before the project begins; to be called upon and reactivated during the project</td></tr><tr><td>Operational Know-How</td><td>To know how to design a standardized relational database, including:<br/>- Building a data dictionary<br/>- Finding functional dependencies<br/>- Applying the standardization process<br/>To know how to create and query this database<br/>To know how to write technical documentation</td></tr><tr><td>Cognitive Know-How</td><td>To know how to manipulate graphs, including:<br/>- Tracing a graph<br/>- Calculating a transitive closing<br/>- Calculating a minimal cover<br/>Know-how is initiated during the mathematics practical work sessions and demanded by the nature of the project</td></tr><tr><td>Social Know-How</td><td>Dependent on group work, including: keeping common documentation, negotiating tasks, respecting roles, directing the work (for the project manager), solving conflicts, etc.</td></tr><tr><td>Public Targeted</td><td>Students with a technological certificate: undergraduate students in a short cycle class.</td></tr><tr><td>Module period</td><td>5 sessions of 1.5 hours in the presence of the teacher<br/>Homework will be necessary in order to respect the schedule for handing in documents</td></tr><tr><td>Constraints</td><td>Few sessions<br/>Few computer rooms available<br/>A large group of students<br/>Students that have difficulties working outside of class meetings</td></tr><tr><td>Strengths</td><td>Students who have a good mastery of computer tools</td></tr><tr><td>Teacher Role</td><td>Orchestrator / Facilitator<br/>Will perform an audit for each group at each meeting (monitoring progress, providing feedback, answering questions, reactivating motivation if necessary)</td></tr><tr><td>Type of pedagogy</td><td>Collaborative project (5 to 6 learners per group)</td></tr><tr><td>Resources Given</td><td>A "learner kit" to provide guidance, including<br/>• document models,<br/>• a project request form,<br/>• a requirements document that lays out the project's objective and its scope, and<br/>• the teaching unit scenario, which provides the teaching unit objective, activities and deliverables, evaluation modalities, and instructions.</td></tr><tr><td>Kind of Process (free, guided...)</td><td>Semi-free organization (Intermediate dates set by the teacher)</td></tr><tr><td>Evaluation mode</td><td>Evaluation of the returned documents, the executed slides, the group organization according to a criteria grid provided to the learners prior to the start of the project</td></tr><tr><td>Working method</td><td>The learners receive a case study to be handed in 5 weeks.<br/>The teacher sets the dates at which he/she recovers the documents that must be handed in (Reports, schedules, meeting audits, etc.)</td></tr><tr><td>Work Organization (In the presence of the teacher / outside of the classroom / mixed)</td><td>Face-to-face work during the meetings is coupled with homework to be completed outside of the classroom.<br/>A website will make it possible for the teacher to be kept informed of the progress, which is noted before each meeting. The learners will leave records of their activities and upload documents.<br/>During and outside of the practical work sessions, learners will have access to work rooms for their meetings and rooms equipped with computers.</td></tr><tr><td>ICT Instrumentation</td><td>A website per project group containing project progress reports, as well as the project documents, such as reports and meetings audits. Email tool.</td></tr></table>
In the targeted device, the learners have to work collaboratively to produce a
standardized database (Figure 2).

<figure><img src="image_3.png" /><figcaption>Fig. 2. A schematic presentation of the educational setting for case 1</figcaption></figure>

The learners form groups of 5 to 6 people. They designate roles both at the project management level (project manager, writer, communications manager or project librarian) and at the production level (analyst, database administrator or tester). Assisted by their teacher, they follow the principal recommendations concerning, for example, the project duration or the deliverables.

A kit that is distributed to the learners contains useful resources for helping them to set up a project management plan in a university context. It describes tricks and tips for organizing the project. A collaborative site (or a Weblog) insures the communication around the project and also plays the role of library. This site makes group members feel that they are truly a part of the project, which is one of the factors of success. The teacher is supposed to be able to communicate easily with the learners and thus is kept informed about the project's progress. Talon, Toffolon and Warin (2005 and 2007) have provided a detailed presentation of this method, which is represented schematically in Figure 2.

5.1.3 Evaluation Outcome

In previous years, the teacher engaged only technical exercises during the sessions to introduce them to technical standards. She decided to organize her class to insure a greater participation of the learners in their training. She used the tool to build the pedagogical device and completed the evaluation questionnaire after its use.

The final assessment shows a good mastering of the domain skills acquisition (i.e., database normalization) with a higher score on this subject than previous years. (The students performed the same exercise as in the years Y-2 and Y-4.)
In terms of satisfaction with the tool, the teacher appreciated its help to get her thinking. But she noticed a lack of clarity on certain items. She thinks that a stronger guidance (e.g., examples) would be useful to help her make choices.

She evaluated the device developed by taking notes of any learner interventions, examining the standardized databases produced and analyzing the questionnaire filled out by the learners at the end of the teaching unit and before scoring. In the end, 73% of learners who completed the protocol had the impression of having acquired a work method.

Learners emphasized the "attractive" aspect of project-based teaching. They highlighted their motivation, and specified that it does not come "just from marks, but simply from pleasure of providing a quality product." Their training was a hybrid format (in the presence of the teacher and at distance) and offered them the opportunity to meet during in-presence sessions, which, in their eyes, became rather quickly sessions during which they could take stock of their project. An average of 3.27/5 was given for their satisfaction level with the teaching unit, and an average of 3.85/5 given for their comfort level with the proposed pedagogy. The following are some quotations from the learner questionnaires: "The module puts us in the same conditions as in a project in the workplace. It is therefore a very good way to prepare us to work."; "This project was very rewarding because it gave me a clearer picture of project management and a first approach to the work world, even if it was an exercise".

In the end, 73% of learners who completed the protocol had the impression that they had acquired a work method. However, learners required more sustained intervention by the teacher, in the form of discussion with the group. Collaboration among learners led to conflicts in about 50% of the projects. Nonetheless, they felt that work had essentially been distributed equitably; they did not feel that they had done more work than others, nor did they feel they had done undesirable tasks (70% of the responses).

To summarize these analysis results:

- There is a greater motivation among learners and an absenteeism rate near zero;

- There is a real interest in using light ICT solutions (e.g., email, chat rooms, blogs);

- It is important to make project management resources continuously available;

- Collaboration must be fostered through the teaching method and the integration of collaborative tools;

- The teaching method associated with the pedagogical device was valued and is perceived as efficient.

Obviously, the teacher had a greater workload. She had to intervene more, to help advance the project and resolve conflicts. Conflicts are a normal part of working together, but should be controlled by the teacher. She now wants to use more efficient and more rapid mechanisms (e.g., voicemail, broadcast messages reminders or information podcasts).

## 5.2 Case 2: a role-play approach for evaluating interactive systems (websites)

### 5.2.1 Context

The second author of the article (hereafter referred to as author 2) has observed—in his personal experience with ergonomics, during on-site internship monitoring visits and when executing industrial contracts—that the ergonomic evaluation of the interactive systems is often neglected, with the industrial and academic worlds sometimes being completely unaware of the need for this kind of evaluation. Moreover, often, even when evaluations are carried out, they are quite inadequate because they are realized with very general tools, such as checklists, evaluation grids
and questionnaires. The limits of these tools are primarily due to the subjectivity of their use. He thus decided to use the tool to determine a new way to teach the evaluation of interactive systems. He produces the following device.

### 5.2.2 The device

For this reason, author 2 must first demonstrate the limits of these tools to his learners during their first practical application. Then, he approaches ergonomic evaluation, using the Cognitive Walkthrough (CW) method described below.

The CW method is based on the learning-by-exploration model proposed by Polson and Lewis (1990), which was itself inspired by Norman's action theory (1986). It has been object of many variants and evolutions [Mahatody, Sagar and Kolski, 2010] and is used in many companies. The principles of the learning-by-exploration theory are anchored in the following cycle of resolution: Initial goal - action plan- action execution - result evaluation - revision of objectives. The goal of CW is to simulate the user's cognitive path when using the interface [Huart, Kolski and Sagar, 2004]. Users integrate their perception of the system, using their basic knowledge to build a representation that will allow them to carry out a task. A form containing specific questions guides the user's path. The method proceeds in two phases for each task: during the preparation phase, the evaluator describes the interface's initial state, the sequence of actions used to carry out the task and the user's goals; during the evaluation phase, the evaluator conducts an in-depth analysis of the human-machine interaction, with the questions on the form guiding the verification of the possible errors in the exploration of the operating model. He summarized the pedagogical device in the following table (Table V).

<table><caption>Table V: A device for developing competences through role-play</caption><thead><tr><th colspan="2">A role play approach to acquire skills for evaluating interactive systems<br>A device for the ergonomics teaching unit</th></tr><tr><th>Skill targeted by the teaching unit</th><th>Knowing how to evaluate an interactive system (website) using the Cognitive Walkthrough (CW) evaluation method</th></tr></thead><tbody><tr><td><strong>Knowledge</strong></td><td>- Theoretical notions of the ergonomics of interactive systems and the Web<br>- The different website design stages<br>- The different website evaluation methods<br>Transmitted through a theoretical presentation (i.e., a lecture illustrated by website design and evaluation case studies) before the role-playing project begins</td></tr><tr><td><strong>Operational Know-How</strong></td><td>To know how to realize the specifications for a website<br>To know how to design and evaluate an HCI based on the dynamic of the actions.<br>To learn the difference between subjectivity and objectivity in ergonomic evaluation.<br>Know-how is required by the nature of the project</td></tr><tr><td><strong>Cognitive Know-How</strong></td><td>To know how to complete the problem solving cycle in order to carry out a well defined task, including:<br>- Finding solutions to the problem starting from a goal,<br>- Making an action plan,<br>- Executing the actions,<br>- Evaluating the result,<br>- Revising the objectives.</td></tr><tr><td><strong>Social Know-How</strong></td><td>Dependent on paired work, including accomplishing the tasks required by the supervisor in precise order, filling in the evaluation card (figures 4a) and problem description card (Figure 4b), writing a common report, and respecting the assigned roles.</td></tr><tr><td><strong>Public Targeted</strong></td><td>Master's (fifth-year) learners in Software and Human Engineering</td></tr><tr><td><strong>Module period</strong></td><td>One 3-hour session in the presence of the teacher<br>Each pair of learners then has one week to write and hand in a report</td></tr></tbody></table>
<table><tr><td>Constraints</td><td>Few sessions</td></tr><tr><td>Strengths</td><td>Learners who have a good mastery of web tools</td></tr><tr><td>Teacher Role</td><td>Orchestrator / Facilitator<br>Active when explaining the steps of the evaluation method, fairly passive during the application of the method.</td></tr><tr><td>Type of pedagogy</td><td>Role-playing, with one supervisor and one evaluator per learner pair</td></tr><tr><td>Resources Given</td><td>Handouts explaining the steps of the evaluation method and the three cards (figures 4a, 4b and 4c)</td></tr><tr><td>Kind of Process (free, guided...)</td><td>Guided for the actions and free for the answers and the evaluation</td></tr><tr><td>Evaluation mode</td><td>Evaluation, correction and grading of the report handed in.</td></tr><tr><td>Working method</td><td>The learners receive the handouts and read them. The teacher then explains the method, helps the supervisor choose the actions needed to accomplish the task. The teacher supervises the role-playing session, which lasts three hours, and corrects the reports.</td></tr><tr><td>Work Organization (In the presence of the teacher / outside of the classroom / mixed)</td><td>The role-playing is done in the presence of the teacher in a practical workroom equipped with computers and Internet access. The cards are filled in during this 3-hour session. The reports are written outside the classroom by the learner pairs and are handed in one week later.</td></tr><tr><td>ICT Instrumentation</td><td>Use of computers and navigation on professional websites in order to apply the CW method.</td></tr></table>

Using an approach (the device is summarized in figure 3) that tries to put learners in situations that are close to reality, each pair of learners takes turns playing the roles of supervisor (designer) and evaluator (user). Thus, with each learner playing one of the roles, the learners complete an ergonomic evaluation using the CW method. The learners playing the role of the designer first break down the task into actions that must be taken to achieve their goals. These goals are supposed to be representative of a typical interactive software use. They then fill in card 1, called "Evaluation Preparation" (Figure 4a). The sequence of actions written on the card is submitted, action-by-action, to the learners playing the evaluator role. Once the first action has been evaluated, the evaluators move to the second action and so on. The learner designers thus supervise the learner evaluators, in order to answer any questions and intervene if there is a problem, but also in order to be able to critique the use of the method in the simulation report, which is written by both learner partners.
<figure><img src="image_4.png" /><figcaption>Fig. 3. Device for Interactive System Evaluation</figcaption></figure>

The learners playing the role of evaluator simulate the system users. They apply the sequence of actions transmitted by the supervisors, filling in card 2, called “Evaluation” (Figure 4b). For each action requested by the supervisors, the evaluators must note the required task (e.g., “book a train ticket”) and the action to be taken (e.g., action 1 “type voyage-sncf.com”). They then evaluate this action by passing obligatorily through the following steps: (1) before the action, they must imagine the goal and write it down on card 2 (e.g., what information must be found before consulting the screen page: menu, icon, image, invoice, etc.); (2) during the action, they consult the page and answer questions 2.1 to 2.3; and (3) after the action, they answer questions 3.1 to 3.4. They must fill as many evaluation cards as there are actions submitted by their supervisors. For each action, they answer the questions suggested by the CW method on the evaluation card. If they encounter problems, they fill out card 3, called “Problem Description” (Figure 4c).
<figure><img src="image_5.png" /><figcaption>Fig. 4. (a) Evaluation Preparation Card, (b) Evaluation Card, (c) Problem Description Card</figcaption></figure>

The cases treated by the groups only concern real situations: they have to evaluate professional e-commerce websites, to identify their usability problems and propose solutions to solve them, as in industry. For many websites, the number of usability problems can be very high (dozens of problems can be identified for well-known commercial sites).

### 5.2.3 Evaluation Outcome

Author 2 applied the tool to design a pedagogical device for 25 Master's learners in Software and Human Engineering at the University of Valenciennes. The pedagogical approach previously used was based on theoretical courses and practical work, using conventional methods in ergonomics, such as questionnaires and checklists. This subjective approach has many limitations. Filling in the table, the teacher chose to be closer to reality, designing a device in which the learners play a role to place them in the position of users. This method can identify the user's cognitive activity by focusing on his/her current actions and not on subjective responses from the traditional questionnaires and checklists.
The teacher thought that, using the tool, he produced a satisfactory device. Furthermore,

- Using the evaluation grid is very helpful for analyzing the advantages and the disadvantages of the pedagogical approach in order to improve it, but requires more rigor, particularly at the organizational level to receive good feedback;

- Learners must be willing to focus on the past situation to move closer to the real situation, while analyzing retrospectively their own work;

- The tool allows the teacher to step out the conventional teaching framework and get as close as possible to the real situation;

- It would be best to use this tool once the educational approach has progressed so that it does not get out of context.

To assess the pedagogical device designed, the evaluation process used:

- Interaction with learners in the classroom (e.g., questions, comments),

- Feedback from the learners using the device,

- Handouts explicitly requiring a critique of the device,

- Reports from learners had used the device.

Learners really enjoyed the tasks requested online and the ability to freely propose solutions to the problems encountered. The comments in the reports summarize learner opinions:

- 85% reported that they found the device was a move in the right direction in terms of the previously used classic questionnaires and checklists.

- 30% noted that, over time, the device became increasingly cumbersome to use and significantly reduced their ability to pay attention. They proposed introducing a break every hour to avoid fatigue and learner disconnection.

- 55% felt that the issues raised in the grids became increasingly redundant and required more and more concentration.

- 20% preferred a device that would allow them to respond freely, critiquing the sites without being constrained by any medium or grid ("surf and react spontaneously to some extent!").

The teacher did not notice any major conflict between learner pairs because work was always fairly well distributed, and each learner only took care of his own tasks, as written in the support handout.

## 5.3 Case 3: a competitive project-based approach for specifying supervision HCI in complex systems

### 5.3.1 Context

The third author of the article (hereafter referred to as author 3) has observed—in his personal experience with interactive system analysis and design, during on-site internship monitoring visits and when carrying out industrial contracts—that many analysis and design errors are committed by the designers of human-computer interfaces (HCI) in complex systems [Kolski, 1997]. In fact, he thinks that HCI design needs new, more appropriate pedagogical methods. He used the tool to produce the following device.
### 5.3.2 The device

The teacher used the tool to re-design the HCI Specification module, intended for Master's learners. The result of the HCI Specification module is summarized in Table VI.

<table><caption>Table VI: A device for developing competences through competition</caption><thead><tr><th colspan="2">A competitive project-based approach for specifying supervision HCI in interactive systems</th></tr><tr><th>Skill targeted by the teaching unit</th><th>Knowing how to write specifications for supervision HCI in interactive systems</th></tr></thead><tbody><tr><td>Knowledge</td><td>- Basic concepts about the human operators in control rooms<br>- Traditional and advanced information presentation methods and assistance systems used in control rooms<br>- Reactions of human operators with respect to the available HCI<br>- General knowledge about the organization and the structure of control room informational supports<br>- Knowledge related to the on-screen presentation of information Transmitted through a theoretical presentation</td></tr><tr><td>Operational Know-How</td><td>To know how to specify a HCI to respond to a customer request, including<br>- Analyzing an industrial process and highlighting the important information for the human operators in the control room<br>- Proposing an architecture for supervision HCI<br>- Proposing the appropriate modes of representation for each screen page<br>- To know how to write technical documentation (e.g., the response to an invitation to tender)<br>Know-how is required by the nature of the project</td></tr><tr><td>Cognitive Know-How</td><td>- To understand each subsystem and each variable of the industrial process in order to comprehend the incidences and necessities related to HCI<br>- To be able to compare different working hypotheses and sometimes contradictory ideas within the group, with the priority being meeting the control room operators needs<br>Know-how is initiated during the guided work sessions in automation and ergonomics and required by the nature of the project.</td></tr><tr><td>Social Know-How</td><td>- For the group work: providing a common specification document, negotiating tasks, respecting roles, managing (for the project manager), solving conflicts, etc.<br>- For the customer relationship: designing a supervision HCI that meets the customer's needs.</td></tr><tr><td>Public Targeted</td><td>Master's learners</td></tr><tr><td>Module period</td><td>5 sessions of 2 hours in the presence of the teacher, one per week (4 group specification sessions, 1 sales presentation session). Homework is generally necessary in order to respect the schedule for handing in the dossier at the beginning of the last meeting</td></tr><tr><td>Constraints</td><td>Few sessions</td></tr><tr><td>Strengths</td><td>Learners who have a good knowledge in industrial systems</td></tr><tr><td>Teacher Role</td><td>Facilitator / Controller / Customer<br>Provides an assessment for all the groups during the last meeting, including giving feedback, answering questions.</td></tr><tr><td>Type of pedagogy</td><td>Role-playing with competition (4 or 5 learners per group)</td></tr><tr><td>Resources Given</td><td>Handouts (distributed progressively)<br>The invitation to tender, including the description of the industrial process that must be supervised using a control room HCI.</td></tr><tr><td>Kind of Process (free, guided...)</td><td>Semi-free organization (at each meeting, the teacher checks the progress of each group)</td></tr><tr><td>Evaluation mode</td><td>Evaluation during the last session after the sales presentation, based on the quality of the dossier handed in, the quality of the presentation, and the ranking of the groups</td></tr><tr><td>Working method</td><td>After the theoretical lecture, the learners receive the invitation to tender.<br>They must turn in their dossier in 4 weeks.<br>The teacher plays the role of a human operator and answers the</td></tr></tbody></table>
<table><tr><td></td><td>learners' practical questions during the first four sessions. During the fifth and last session, in addition presenting the HCI prototype, the learners also play the part of the selection committee of the groups.</td></tr><tr><td>Work Organization (In presence of the teacher outside of the classroom / mixed)</td><td>Work in the presence of the teacher (the first 4 specification sessions) is coupled with work outside of the classroom The learners have access to work rooms for their meetings and rooms equipped with computers outside of the specification sessions.</td></tr><tr><td>ICT Instrumentation</td><td>The majority of the groups used professional software for creating mock-ups or prototypes of interactive applications.</td></tr></table>

The project concerns a response to an invitation to tender for a Human-Computer Interface (HCI) and uses competition to motivate the learners. For this project, learners are organized in groups of 4 or 5, with each group simulating a company that is responding to the invitation to tender. The project is divided in three phases, each concentrating on a different aspect of the design process. The first is a preliminary phase consisting of 10 hours of theoretical presentations about the problems encountered when designing supervision HCI, using examples of HCI from real industrial projects. The second is a research and development phase. During this second phase (2 hours/week for 4 weeks), the learners work on their response to the invitation to tender. The third and final phase is devoted to sales, each group trying to convince the selection committee that its prototype is the best.

The second phase consists of four 2-hour sessions, on per week for 4 weeks. All the groups work in the same classroom, each located at a distance sufficient to insure the confidentiality of their discussions. The device is summarized in Figure 5.

<figure><img src="image_6.png" /><figcaption>Fig. 5.: A competitive project-based approach</figcaption></figure>

The competing companies must produce a HCI prototype intended to supervise a group of 5 mixing stations (in chemistry industry); a control room operator working with several roundsmen will use this HCI. Each group must also propose an installation map of the control room and a first mock-up of the interactive systems. Extracts of representative mock-ups proposed are visible on figure 6; most of the mock-ups proposed are really professional. During these four sessions, the learners can ask the teacher (the Game Master) discrete questions. The Game Master plays
the role of the human operator (customer) made available to the companies participating in the invitation to tender.

<figure><img src="image_7.png" /><figcaption>Fig. 6. Two examples of proposed mock-ups</figcaption></figure>

During the final phase, each group must make a sales presentation, answering questions about their prototype and responding to constructive criticism offered by the selection committee, played by the teacher and the members of the other groups. In order to insure that no group will "plagiarize" the solutions presented by the other groups, only the elements in the dossier handed in at the beginning of session can be presented. The companies are then ranked, and an audit of the positive and negative aspects of the propositions is then posted.

### 5.3.3 Evaluation Outcome

This competitive project-based approach was used with 25 Master's learners in Electronic and Industrial Software Engineering at the University of Valenciennes. Previously, the teacher gave very specific exercises that were unrelated to industrial
problems and project organization, which was somewhat challenging for learners. He decided to put them in a project situation, close to a real business situation, and to place the learners in competition. He completed the evaluation questionnaire after using the tool.

In terms of his satisfaction with the tool, he thinks that it was useful for asking new questions about the teaching units and their evolution. He also thinks that the tool could provide additional details in the form of examples, but giving too many details would make teachers feel that their approach to evolving modules was too controlled. In addition, he feels that the tool allowed him to think out of the box, or in other words, outside the framework of conventional teaching.

The assessment of the pedagogical device was based on:

- The interventions of learners

- The analysis of the HCI specification documentation provided by each group of learners.

Learners commented on the motivation engendered by being in conditions close to the industrial reality and producing professional quality results (i.e., a prototype of human-computer interface). Each group gave an oral presentation in front of all the other groups, this time playing the role of evaluator. The groups submitted a written report on their prototype. They also participated in a competitive process, forcing them to go deep into the solutions they proposed, caused by rivalry between groups. Nonetheless, the teacher emphasizes the importance of staying highly concentrated in the class meetings. He had repeatedly to switch roles to avoid temporal losing time in the meetings, which may affect the overall approach.

This device led to very good results, as shown by the evaluation of the HCI specification documentation, using eight groups of criteria (i.e., room layout, graphics and representation of the information on screens, the information available on the screens, alarms, communications, time management, linked options and proposals, and work suggested approach), for a total of 35 criteria detailed by Kolski, Sagar and Loslever (2004a and 2004b). The analysis shows that the learner groups typically ignore none of the criteria. This is a very important point for future HCI designers in industrial companies [Kolski, Loslever and Sagar, 2012].

The teacher did not notice any major conflict in the groups or between groups, as work was still fairly well distributed among the groups, both during the work sessions and the final oral presentation of the work. Unanimously, the learners express satisfaction at having gained skills that will serve them in the real world of business.

## 5.3 Observed changes in teaching practices

As a result of observing the devices, a Venn diagram was constructed showing the intersections of the three teaching methods used in the three areas of interactive systems. Figure 7 shows how the key concepts developed in our teaching intersect and overlap. As shown in Figure 7, the Venn diagram highlights key concepts connecting the various training methods for developing competences. It shows that all three teachers used active pedagogy (e.g., project-based pedagogy) under different forms. All three teachers previously practiced traditional teaching sessions with lectures followed by tutorial sessions to practice exercises and practical sessions where students implemented exercises on computers.
<figure><img src="image_8.png" /><figcaption>Fig. 7. Venn diagram of the key concepts connecting the various training methods for developing competences</figcaption></figure>

In each of 3 experiences, the following elements could be observed:

(1) The organization of classroom time has evolved. Despite the inflexible university structure, teachers have been able to establish a more flexible organization for the work sessions, adopting the rhythm that they wanted for the project.

(2) The learner-teacher relationship has taken on a completely different form. The learners have become active participants in their learning, questioning, asking for advice, negotiating.

(3) The groups of learners are more animated and more pleasant and have a stronger desire to succeed because of the changed learner-teacher relationship. The learners obviously find more "pleasure" in their work: there is a relaxed but concentrated environment in the classroom; the learners work steadily and work between meetings; they have a feeling of belonging to a team and of rising to a challenge.

(4) The classroom "landscape" has evolved. When possible, the tables are arranged in a manner that will facilitate discussion.

(5) The tasks are assigned in order to finish the project. The context is no longer "identical tasks for everyone".

(6) Since the evaluation method is announced at the beginning, it is accepted by the learners. A winning attitude can be observed, one that does not look only at the grade but rather at the quality of the work.

(7) The teachers' attitudes have also changed. They have easily adopted their new roles of advisor, controller and facilitator. The learners now take their teacher's advice more seriously. The way that the teachers see the process leads them to distribute information in a different way—handouts, discussions, references to articles—and at a different rhythm.

(8) The teachers also find more "pleasure" in their work. They participate differently, more actively in the construction of knowledge. They are better
able to discover the know-how, errors and evolving competence of each learner.

(9) The learners have great expectations in terms of their work and truly feel that they are learning because they work in conditions that are very close to reality. Far from an inflexible system, they must measure, test and identify their level of know-how. A questionnaire given at the beginning of module systematically highlights their expectations for these projects: "I will learn how to manage a project.", "I will learn how to test a software".

However, implementing this active pedagogy is not easy and the following difficulties are observed:

(1) There is a vast amount of preparatory work: the process must be programmed precisely and the resources must be ready, as they are needed, "on demand".

(2) Evaluation is a permanent process. After each meeting, it is necessary to monitor the work done and the possible problems. This implies that the teachers schedule one period for evaluation and make the time to do it. Our university structures have some difficulty recognizing this background work

(3) Our university structure is not yet really appropriate for implementing active pedagogies. Teachers must deal with the rooms and schedules assigned and negotiate with the administration, sometimes accepting imperfect work conditions.

(4) It is essential to have a good command of the subject in order to adopt an active method that GRASP suggests. This mastery allows the teacher to answer the endless learner questions and to provide them with appropriate resources and advice. This kind of organization can pose problems for teachers who lack experience.

## 5.4 Evolution of GRASP

The preliminary evaluation showed that GRASP is a tool that helps teachers to design pedagogical devices. It has allowed us to move from a traditional pedagogy to an active pedagogy. All three teachers have designed and implemented a project-based pedagogy with the differences shown in the Vienn diagram (Figure 7). The teachers appreciated the help of GRASP to get their thinking on track. However, two of the teachers noticed a lack of clarity on certain items and thought that a stronger guidance (e.g., more examples) would be useful for helping to make choices. Learners appreciated the three devices. They showed more enthusiasm and were more implicated in their work. The analysis of the deliverables showed that the production was good quality.

This analysis made it possible to validate the advantages of the pedagogical devices to achieve higher quality domain and transverse competences. The decision to perfect the GRASP tool and then perform a new evaluation was taken. According to the remarks of two users, GRASP was perfected by adding examples (in red in Table IX in the appendix) and stating several entries precisely. Global contextualization entries were added (in blue in Table IX in the appendix) to allow an immediate description of the teaching unit and its general conditions (i.e., to set the limit for the teaching unit). Aid texts were also improved in spots, according to the remarks the teachers made in the questionnaires.
# 6. THE EVALUATION PROCESS OF GRASP, VERSION 2

## 6.1 Study Object

A larger population of teachers can use GRASP V2. GRASP is a tool that is fast and easy to use, which helps teachers to develop pedagogical devices tailored to their professionalization goals with more active learning activities.

## 6.2 Study Instance:

Teachers: 15 teachers at three universities – 9 teachers in the interactive systems domain and 6 teachers in other domains (Civil Engineering, Mechanics, Electrical Engineering, Information and Communication Sciences, Psychology). See Table VII for profiles.

<table><caption>Table VII: Teachers' profiles</caption><thead><tr><td>ID</td><td>Sex</td><td>Age</td><td>Teaching Experience</td><td>Domain</td><td>Teaching Unit</td></tr></thead><tbody><tr><td>1</td><td>F</td><td>33</td><td>7</td><td>Interactive systems</td><td>UML</td></tr><tr><td>2</td><td>F</td><td>45</td><td>20</td><td>Civil Engineering</td><td>Work Pathologies</td></tr><tr><td>3</td><td>M</td><td>30</td><td>8</td><td>Interactive systems</td><td>Database Management Systems</td></tr><tr><td>4</td><td>M</td><td>54</td><td>24</td><td>Information and Communication Sciences</td><td>Learning and Teaching Strategies</td></tr><tr><td>5</td><td>M</td><td>46</td><td>22</td><td>Interactive systems</td><td>Artificial Intelligence: Logic Foundations</td></tr><tr><td>6</td><td>F</td><td>40</td><td>22</td><td>Interactive Systems</td><td>Advanced UML</td></tr><tr><td>7</td><td>M</td><td>48</td><td>24</td><td>Interactive Systems</td><td>Requirement Analysis</td></tr><tr><td>8</td><td>M</td><td>56</td><td>26</td><td>Psychology</td><td>Multimedia System Design and Evaluation</td></tr><tr><td>9</td><td>F</td><td>48</td><td>25</td><td>Interactive Systems</td><td>Interactive System Design</td></tr><tr><td>10</td><td>F</td><td>48</td><td>23</td><td>Interactive Systems</td><td>Operating Systems</td></tr><tr><td>11</td><td>F</td><td>45</td><td>19</td><td>Interactive Systems</td><td>Networks</td></tr><tr><td>12</td><td>M</td><td>34</td><td>11</td><td>Electronical Engineering</td><td>Numerical Transmission</td></tr><tr><td>13</td><td>M</td><td>35</td><td>13</td><td>Electronical Engineering</td><td>Monitoring Measurement and Data Acquisition Instruments</td></tr><tr><td>14</td><td>F</td><td>37</td><td>8</td><td>Interactive Systems</td><td>Object Design</td></tr><tr><td>15</td><td>M</td><td>51</td><td>30</td><td>Mechanics</td><td>Solid Mechanics</td></tr></tbody></table>

The public has been split into two groups: a group of interactive systems teachers and a group of teachers in other domains. The will was to verify if the use of GRASP could be extended to other teaching units and domains and which ones.

Environment: ULCO, UVHC, UPJV

Time and place: September to November 2011 at all Universities (ULCO, UVHC, UPJV).

## 6.3 Tools and techniques tested

Grasp V2.

## 6.4 Learning and teaching theories

Behaviorist, Cognitivist and Constructivist theories are possible. However, we recommend active learning forms that encourage competence acquisition.

## 6.5 Educational Domain

Interactive systems, or technical domains making intensive use of interactive systems.

## 6.6 Study approach

Fifteen teachers used GRASP to design or redesign a device appropriate to their needs and constraints in their teaching units. An evaluator was at their side to
answer eventual questions and to make choices explicit according to the explicitation interview (see section 6.7). For each interview, the following technique has been adopted:

- The evaluator presents the aim of the evaluation and its modalities. He/she presents the tool and its structure (20 minutes).

- The teacher fills in the tables in the presence of the observer. He/she can ask questions to clarify the entries. The Observer notes questions and remarks (1h to 2h30). The observer asks for explanations after the filling in the cells.

- The teacher fills a questionnaire about the usability of the tool and its advantages (30 minutes to 1h00).

## 6.7 Data collection

The following data collection technique was carried out:

- For interviews, the explicitation interview [Vermersch, 1994; Maurel, 2009] was chosen. The explicitation interview is a technical verbalization support, which takes place once the work has been performed. The aim is to make the teacher explain the way he/she behaves, with a verbalisation. The teacher's choice is not influenced. This interview technique was chosen in order to obtain feedback on the actions.

- A teacher questionnaire about the tools' usability and the arrangements of the designed device.

## 6.8 Data analysis

A research approach based on statistical information collected from the teacher (i.e., satisfaction rate, difficulties) was applied. This approach was implemented by analyzing the data collected via the questionnaires and interviews.

## 6.9 Study Outcome

The overall results of the evaluation are provided in section 7.

# 7. TOOL EVALUATION – PHASE II - OUTCOMES

The second evaluation concerned the modified table (see the appendix), which applied the remarks extracted from the preliminary evaluation, concerning the design, development and use of pedagogical devices in three teaching units. Section 6 described the second evaluation. Its purpose was to test the usability of the table with a larger population, detect faults in the table, and observe the difficulties encountered while the teachers were filling in. The interest of the second evaluation was to analyze the devices designed by teachers, observe the level of difficulty of the design task and obtain constructive criticism from teachers. We analyzed the observer's notes and the teachers' questionnaires. The following section presents the important observations that were made.

## 7.1 Results

The first step was to produce a synthetic view of the design devices (Table VIII).
<table><caption>Table VIII: A synthetic view of the devices</caption><thead><tr><td>Id</td><td>Teaching Unit</td><td>Global Orientation of the Designed Device</td></tr></thead><tbody><tr><td>1</td><td>UML</td><td>Collaborative Project-based pedagogy – Teacher controls delivery dates of the desired deliverables</td></tr><tr><td>2</td><td>Work Pathologies</td><td>Project-based pedagogy; Simulation; Games</td></tr><tr><td>3</td><td>Database Management Systems</td><td>Problem-based pedagogy and Project-based pedagogy</td></tr><tr><td>4</td><td>Learning and Teaching Strategies</td><td>Simulation and problem-based learning</td></tr><tr><td>5</td><td>Artificial Intelligence: Logic Foundations</td><td>Problem-based pedagogy (challenge)</td></tr><tr><td>6</td><td>Advanced UML</td><td>Collaborative Project-based learning</td></tr><tr><td>7</td><td>Requirement Analysis</td><td>Project-based pedagogy in a group</td></tr><tr><td>8</td><td>Multimedia System Design and Evaluation</td><td>Technical exercises and small projects</td></tr><tr><td>9</td><td>Interactive System Design</td><td>Technical exercises and small projects</td></tr><tr><td>10</td><td>Operating Systems</td><td>Technical exercises and a small project</td></tr><tr><td>11</td><td>Networks</td><td>Concrete case studies during the course; Problem-based learning with a logbook.</td></tr><tr><td>12</td><td>Numerical Transmission</td><td>Problem-based pedagogy; Project-based pedagogy; Games</td></tr><tr><td>13</td><td>Monitoring Measurement and Data Acquisition Instruments</td><td>Several small projects, with students taking part in the evaluation process</td></tr><tr><td>14</td><td>Object Design</td><td>Project-based learning</td></tr><tr><td>15</td><td>Solid Mechanics</td><td>Problem-based learning</td></tr></tbody></table>

The GRASP table was used to identify the differences in the two populations. The differences seem to be between technical, which requires developing professional skills, and non-technical, more focused on developing theoretical knowledge. The teacher of the Teaching and Learning Strategies Unit signaled his lack of interest in certain parts of GRASP, as he is more interested in the logic of knowledge transfer. Overall, teachers felt that the course was moderately motivating for students with the pedagogy they currently use (Cursor 1 - Figure 8). They find generally their teaching motivating for them (Cursor 2 - Figure 8). In their comments, they state they are still mainly followers of the transmissive pedagogy, which is traditional in France. This pedagogy takes control and insures their authority over the students. Eleven of them describe a transmissive organization, with structured teaching units that break down into Lectures/Tutorial sessions/Practical sessions. Twelve teachers report that they use a non-innovative pedagogy, which provides a development of domain knowledge considered as fairly satisfactory (cursor 3 - Figure 8). However, it is considered as relatively inefficient to develop domain know-how (cursor 4 - Figure 8) and very inefficient to develop transversal skills (cursor 5 - Figure 8).
1 - Motivating for learners

<figure><img src="image_9.png" /></figure>

2 - Motivating for teachers

<figure><img src="image_10.png" /></figure>

3 - Efficient to develop knowledge

<figure><img src="image_11.png" /></figure>

4 - Efficient to develop domain know-how

<figure><img src="image_12.png" /></figure>

4 - Efficient to develop transverse skills (written and oral communication, etc.)

<figure><img src="image_13.png" /><figcaption>Fig.8. How is your teaching unit in its current form?</figcaption></figure>

Among the constraints of the subject taught, some involve the non-exciting side of their domain, compared to those taught by other colleagues. Many teachers are faced with a lack of student motivation. In their own words, they have to demonstrate more conviction and creativity to stimulate learners. The 15 teachers opted for a makeover of their teaching, with a more active involvement (See Table VIII).

GRASP seems to be relatively easy to use (cursor 1 - Figure 9). During the explicitation interview, two teachers reported that they needed to be assisted and/or have feedback about their choices. Two teachers reported difficulties with the vocabulary, for which they sought clarification (e.g., cognitive skills, operational expertise, procedural know-how). GRASP appears to be fairly well detailed (cursor 2 - Figure 9). It has really helped to clarify the pedagogical expectations (cursor 3 - Figure 9) and somewhat helped to clarify the constraints (cursor 4 - Figure 9). The teachers evaluated the GRASP tool as very useful (cursor 5 - Figure 9).

1 - Is GRASP easy to use?

<figure><img src="image_14.png" /></figure>

2 - Details in the GRASP table

<figure><img src="image_15.png" /></figure>

3 - GRASP has helped you to clarify your pedagogical expectations

<figure><img src="image_16.png" /></figure>

4 - GRASP has helped you to clarify your constraints

<figure><img src="image_17.png" /></figure>

5 - Do you find GRASP useful?

<figure><img src="image_18.png" /><figcaption>Fig. 9: How about GRASP?</figcaption></figure>
Three teachers found some entries that they did not understand; they asked the explanation of these entries and an oral explanation was immediately given. Only one teacher had difficulty identifying the generic skills targeted by the teaching unit. Two teachers struggled to identify the targeted knowledge and know-how. They had no trouble identifying the specifics of their teaching unit. The majority of teachers appreciated the assistance given in the form of examples. The GRASP pedagogy was isolated and the resources were identified without any difficulty. Many teachers were inspired by the elements of resource classification presented in an example in order to determine the elements that they want to make available to students. Teachers easily identified their role. Figure 10 summarizes all of these statistics (Yes in blue and No in red).

<figure><img src="image_19.png" /><figcaption>Fig. 10. Filling in the GRASP entries</figcaption></figure>
Teachers find that GRASP has allowed them to build a device, which they think is fairly innovative (cursor 1 - Figure 11), but which improves relatively to the previous announced situation. They said that this new device will change a lot their practice (cursor 2 - Figure 11).

1 – With the help of GRASP, do you think that you have designed an innovative device?

<figure><img src="image_20.png" /></figure>

2 – Do you think that the future device will change your pedagogical practice?

<figure><img src="image_21.png" /><figcaption>Fig.11. The designed device</figcaption></figure>

Twelve teachers believe that they have been guided to choosing more active methods. Of these, 9 of them propose to distribute more educational resources for students. Thirteen teachers think they will build and distribute a wide variety of resources. The analysis shows a clear trend towards the diversification of activities. The evaluation process globally diversifies more. There is no clear evolution on this point. Furthermore, the scenarios designed with GRASP promise a stricter organization in most of the devices designed. They almost all start with a complete module presentation. Four teachers had difficulty identifying a learning process. According to the verbalization during explicitation interviews, these persons needed more time to imagine the new process. Thirteen teachers consider the use of ICT as a pedagogical support, especially for document management and communication. Many suggested Moodle already widely available in their respective institutions. Twelve teachers also feel they will use GRASP to design another module, and 11 are ready to re-use this tool to re-design their device. Figure 12 summarizes the relevant statistical data: "Yes" in blue and "No" in red.
<figure><img src="image_22.png" /><figcaption>Fig. 12. Changes in the designed device and re-use of GRASP</figcaption></figure>

## 7.2 Synthesis of the second evaluation

Overall, the analysis shows a definite interest in using GRASP, and it does not highlight difficulties in using this tool. The teacher evaluations show the interest in designing devices, referring to a more active involvement. The value of GRASP is to better design or re-design pedagogical devices. Some teachers tend to design or re-design their teaching units without methodological support, but they sometimes lose
efficiency due to lack of method. The GRASP tool offers them this methodological framework.

Real sentences drawn from the interviews frequently focused on the possibility offered by GRASP for formalization:

"For thinking"; "I was able to verbalize"; "I was able to formalize"; "this table allowed me to place the reason for my attitudes, it helped me write my expectations"; "Table power to think and fill in the details"; "it forces me to think and plan"; "it helps to lift inhibitions related to uses, fully integrated with my behavior"; "it allowed me to unearth ideas that I had had at some point and I did not notice at that time".

The results show that the GRASP tool promotes active teaching methods. In fact, teachers have turned to more active teaching methods and have become aware of the need to assess differently, even if it is difficult. Many project-based pedagogies have been developed in various forms: mini projects, competitions, and different group sizes. This confirms the results obtained in the preliminary evaluation (Figure 7). Like other instructional design methods, GRASP is a tool to use as part of an iterative design cycle; the appropriate device can't be obtained in one cycle.

## 8. CONCLUSION AND PERSPECTIVES

In this article, we have defined some important concepts related to professionalization and have established connections between these concepts and the concept of competence. We then gave a status report of professional competence in the field of engineering, more specifically in the field of interactive systems engineering. We highlighted the weaknesses of the student engineers, particularly in the softer skills, such as communication and project management.

We proposed a manageable support tool to facilitate the development of a pedagogical device. This tool asks a number of questions about the intended learning objectives. In order to validate this tool, a preliminary evaluation was designed which analyzed three educational initiatives in interactive systems domain and their impact was discussed. The first evaluation allowed us to show the usability and the advantages of GRASP.

Then, a second evaluation was performed with a larger population to confirm the preliminary results and to verify the impact on a larger panel of teachers. GRASP was used by 15 teachers, who designed a pedagogical device. The results of this second evaluation were very promising and allowed us to validate the usability of GRASP and to detect errors in GRASP. It has notably shown the necessity to define some concepts, such as operational know-how and cognitive know-how. The explicitation interviews showed that the interaction between the teacher and the evaluator promotes confrontation of ideas that was favorable to appearance of questioning the old methods and moving towards a more relevant innovative education, which better meets the needs of students and teachers alike. In fact, the rich content of the table is enhanced by the explicitation interview's interactions.

In the context of the development of pedagogical devices, Information and Communications Technologies (ICT) are a means of supporting the communication between the students, as well as the communication between students and their teachers. The ICT also offer interesting ways to store and distribute documents. They also can reduce the teacher's workload through centralization. For example, the tools needed for piloting and controlling student work could be centralized on a dedicated website.

Our medium-term goal is to perfect GRASP and produce a Computer-Aid Assistant based on GRASP to better help teachers to design a device adapted to their
context. For this purpose, we are interested in using pedagogical patterns [Labour and Kolski, 2010]. Tripp & Bickelmeyer (1990) suggests that instructional software templates may perfect the development process and compensate for the lack of experience of the developers (i.e., teachers). This can be an interesting direction for the authors of instructional software.

# APPENDIX

<table><caption>Table IX: Grasp V2</caption><thead><tr><th>Method</th><th>Give a name which summarizes your general method</th><th>Acquisition of competences in normalization of Relational Databases</th></tr></thead><tbody><tr><td>Teaching Unit</td><td>Describe the teaching unit concerned by the innovating pedagogy<ul><li>Name</li><li>Total duration</li><li>Number of sessions per week</li><li>Number of teachers</li><li>Aim</li></ul></td><td>Description of the Teaching Unit<ul><li>Database</li><li>16 weeks</li><li>3 allocated sessions per week: 1 session 1:30 with all learners, a 1:30 session with 1/3 group and a 1:30 session in a computer room.</li><li>4 speakers: 1 lecturer, 2 PhD learners and a professional</li></ul>The module aims to Understand<ul><li>What is a relational database</li><li>What is a DBMS</li></ul>The module aims to Know-how<ul><li>Describe and use a DB</li></ul>The module aims to Know<ul><li>Establish and administer a comic</li><li>Optimize DB</li><li>Securing Data</li></ul></td></tr><tr><td>Skill targeted by the teaching unit</td><td>Summarized in one sentence skills covered by the device<br>It must be clear in your mind. Avoid having too many demands simultaneously.</td><td>Learning to design and manipulate a relational database in 3rd Normal Form.</td></tr><tr><td>Knowledge</td><td>Detail knowledge required for this skill – The Know What. List what will allow the learner to think and perform.<br>You must separate the knowledge the learners have prior to the module from the knowledge that will be acquired during the module.<br>You must separate the theoretical knowledge (know what) and the procedural knowledge (know how to).<br>You must answer the following questions: "What knowledge has already been acquired (theory and procedure)" and "What knowledge do the learners need to acquire (theory and procedure)?"<br>Do not forget to cover domain knowledge and transverse knowledge.</td><td>Before applying the teaching method, Learners must have studied and therefore know (other methods used previously)<br><u>Domain knowledge</u><br>KNOW WHAT:<ul><li>The concept of relationship of Database</li><li>SQL language</li><li>Algebraic operators</li></ul>KNOW HOW TO:<ul><li>Instructions to query a database</li><li>Join mechanisms</li><li>Instructions to update and delete data</li><li>Instructions for creating tables in a DBMS</li></ul>After application of the method, they should know<br><u>Domain knowledge</u><br>KNOW WHAT:<ul><li>The reasons and benefits of standardization</li><li>The 3 first normal forms: 1NF, 2NF and 3NF</li><li>Functional dependencies</li></ul>KNOW HOW TO:<ul><li>Recognize a 1NF, 2NF, 3NF relationships</li><li>Move into 1NF, 2NF, 3NF</li><li>Process to a normalization</li><li>Create a Database in a normalized form</li></ul>Transversal knowledge<br><u>KNOW WHAT</u></td></tr></tbody></table>
<table><tr><td></td><td></td><td><ul><li>The stages of a project</li><li>Roles in a project team</li></ul><p><strong>KNOW HOW TO:</strong></p><ul><li>Conduct a project group</li><li>Follow a schedule</li><li>Report on his/her activities</li><li>Write a Business Case</li></ul><p>The knowledge can be transmitted through reading but will be mostly implemented and reactivated by the method</p></td></tr><tr><td><strong>Operational Know-How</strong></td><td>Describe the different procedures (know how to operate) learners must master to be able to act. List the skill set required to execute, reproduce and adapt procedures required by a job.<br/><br/>You must separate the procedures the learners have already mastered prior to the module from the procedures that will be acquired during the module.<br/><br/>You must answer the following questions "What must a learner be able to do and to reproduce?" and "What will a learner be able to do and to reproduce after the module?"</td><td><p><strong>Before applying the teaching method, learners already have being able to:</strong></p><ul><li>Build a data dictionary</li><li>Read and understand a MLD</li><li>Recognize the primary keys and foreign keys</li><li>Handle (query, update) in a SQL DB already established and standardized</li><li>Describe and model the algebraic operations needed to perform a query</li></ul><p><strong>After applying the teaching method, learners need to know</strong></p><ul><li>Create a database in 3NF applying a normalization process</li><li>Find and write functional dependencies</li><li>Identify and justify the normality of a relationship</li><li>Manage a small project and its documentation in group</li><li>The skills can be worked on exercises and reactivated by working on a project</li></ul></td></tr><tr><td><strong>Cognitive Know-How</strong></td><td>Describe the different capacities that learners must master to function: capacities to process information, reason, learn, and solve problems. List the skill set required to acquire knowledge in order to analyze and understand the world, exchange and communicate with others and solve problems<br/><br/>You must answer to the question: "What must a learner be able to do?" and "What will a learner be able to do after the module?"</td><td><p><strong>To develop targeted cognitive skills, learners need (acquired in mathematic module):</strong></p><ul><li>Know how to draw a graph</li><li>Know how to calculate a transitive closure</li><li>Know how to calculate a minimum coverage</li></ul><p><strong>After application of the method, learners will have improved their mastery of these skills and have worked on</strong></p><ul><li>Identification and anticipation of future tasks</li><li>The recovery time of an activity</li><li>The projection over time.</li></ul></td></tr><tr><td><strong>Social Know-how</strong></td><td>Describe the different social capacities learners must master to act in the targeted domain. List the Skill set required to establish reciprocal relationships with positive partners (e.g., giving and taking fairly, developing conditional cooperation) in disciplines implicit in daily life, while implementing such success factors as motivation, anticipation, positive self-image, sense of responsibility and control of space.<br/><br/>You must answer the following questions: "What social qualities are expected for learner to be able to act?", "What social qualities are to be developed by the module?"</td><td><p><strong>Learners must be able to:</strong></p><ul><li>Connect with others</li></ul><p><strong>After application of the module, they must have worked on:</strong></p><ul><li>Negotiation</li><li>Autonomy</li><li>The delegation of tasks</li><li>Respect for others</li><li>Assistance to other</li><li>Conflict resolution</li><li>Respect of their missions</li></ul></td></tr><tr><td><strong>Public Targeted</strong></td><td>Define your public:<ul><li>Level</li><li>Total number of learners</li><li>Number of divisions of the class</li></ul></td><td><p><strong>Public:</strong></p><ul><li>University / Second year / 3rd Semester</li><li>50 learners</li><li>2 groups of 25 learners</li></ul></td></tr></table>
<table><tr><td></td><td><ul><li>Specific difficulties: list the specific difficulties of your public (e.g., agitated, decentralized, etc.)</li><li>Specific advantages: list the specific advantages of your public (e.g., of good will, handles the computer well, etc.).)</li></ul></td><td><ul><li>Large Group => agitated, not completely autonomous</li><li>ICT proficient, almost all learners own a laptop and an Internet connection, wireless terminals at the IUT, like practice.</li></ul></td></tr><tr><td>Unit Period</td><td>Define the possible duration and their constraints<ul><li>Session length</li><li>Session frequency</li><li>Possibility of varying the length or frequency?</li><li>Homework?</li></ul></td><td>Sessions imposed by the timetable and program. We will have a limited time to work on the skills:<ul><li>Two 1h30 sessions per week</li><li>Weekly</li><li>No changes possible</li><li>Work outside the sessions as possible but without having too much involvement on a charge by their increased synthesis projects</li><li>We can spend to better 5-week program of Database</li></ul></td></tr><tr><td>Constraints</td><td>Identify the different constraints under which you work.<br/>List the specific difficulties (e.g., technical, organizational)</td><td><ul><li>Work schedules that increase their load</li><li>Some classrooms not equipped with computer</li><li>Inability to temporarily vary their working time on the module</li><li>No negotiation on possible enhancement of additional work for teachers</li></ul></td></tr><tr><td>Strengths</td><td>Identify the different strengths of your learning environment<br/>List the specific advantages of your teaching environment (e.g., human resources, pedagogical resources, equipment)</td><td><ul><li>Learners pleasant and active so motivated</li><li>Motivation of teachers to increase their skills</li><li>Want to upgrade education</li><li>Looking for involvement and increased activity</li></ul></td></tr><tr><td>Teacher Role</td><td>Define the role of the teacher. Choose among:<ul><li>Strengths: continuous monitoring of the progress of activities; anticipation of questions and problems.</li><li>Participatory: an active presence among the learners facilitates frequent and regular monitoring; anticipates questions and problems.</li><li>Intermediate: Monitoring the activity without interfering in the learners' work.</li><li>Weaknesses: provides assistance for learner work; launches and recovers the deliverables at the end of activity.</li></ul></td><td>Given the constraints set (level of education, agitation, limited time), the teacher wants to emphasize:<ul><li>Active participation: a weekly review of progress, to make available at specific times, schedule defined by the teacher to respect scrupulously assistance to problems, reactivation of motivation.</li></ul></td></tr><tr><td>Type of pedagogy</td><td>Select and justify the choice of pedagogy: type and organization.<br/>It is advisable to encourage forms of active learning in groups that promote social skills learning and mobilize of diverse competences: project based pedagogy, pedagogy based on problem-solving, simulation, games, for example</td><td>We will focus our choice on active learning through group project that will strengthen the skills outlined above.<br/>A mini-project involving the distribution of a case study to be conducted in groups of 3 to 5 people over 5 weeks in accordance with project management indicated by the teacher (steps, work to do, at key delivery available making, etc.)</td></tr><tr><td>Resources Given</td><td>Identify the resources (and their format) you have to provide to the learners:<ul><li>Course materials,</li><li>Advice for carrying out the activities</li><li>Scenario,</li><li>Methods,</li><li>Tools,</li><li>Document models,</li><li>Evaluation criteria,</li><li>etc.</li></ul></td><td><ul><li>Lectures on standardization (why to normalize, different normal forms, functional dependencies, Armstrong rules and standards process)</li><li>Document Templates</li><li>Case Study</li><li>Documents on the steps of project management (MEPULCO)</li><li>Scenario that recalls the objectives of the project, said work to do and recall every meeting, work instructions, criteria for evaluation of the work (methodology, quality of materials, quality of the process).</li><li>Tools for writing and drawing. A tool that allows modeling of entity-relationship models.</li></ul></td></tr></table>
<table><tr><td>Kind of Process (free, guided...)</td><td>Define and justify the type of educational process that you want the learners to implement, in line with the teacher role:<ul><li>Completely guided,</li><li>Completely free (only the outcome is important),</li><li>Intermediate process (timepoint for product delivery).</li></ul></td><td><ul><li>Access to a server to keep a log of activities and submit their deliverables.</li></ul>The process is an intermediate process led by the teacher who fixed delivery dates. The scenario given to the learners fixes the dates of delivery and shows the various tasks to be implemented to achieve this. Learners organize themselves to assess the duration of these tasks and distribute the work to deliver within the specified time.</td></tr><tr><td>Evaluation mode</td><td>Define precisely your evaluation modes:<ul><li>Result evaluation,</li><li>Process evaluation,</li><li>Collaborative work evaluation,</li><li>Documentation quality evaluation,</li><li>Etc.</li></ul><p>Do not forget to connect your evaluation to the skills targeted by the module. The learner must be motivated by the evaluation.</p></td><td>The method of evaluation seeks to promote learners' motivation to finish on time while working on the targeted skills outlined above. So we will specify the learner that the system uses a grid that evaluates:<ul><li>The quality of the technical format (Database obtained and operationalization of the database as follows)</li><li>The quality of the process: the ability to plan, to meet deadlines, to distribute the load,</li><li>The quality of collaboration: the involvement of all, the ability to play his/her role in the team, assistance to others, ability to negotiate.</li><li>The quality of deliverables: form (graphic compliance, misconduct, presentation) and substance (content sense).</li><li>The quality of the work: evaluation of a Slideshow showing the work done, difficulties encountered and the skills acquired.</li></ul></td></tr><tr><td>Working method</td><td>Define the learner and teacher activities precisely.<br/><p>A schedule is necessary if you have chosen a completely guided process or an intermediate process.</p><p>You can produce a guideline that will support your sessions and will be distributed to learners.</p></td><td>Case study given in session 1 and then work distributed in respect of a scenario (see scenario distributed to learners)<br/><p>Learners are grouped in a room and work at their own pace.</p><p>The teacher must prepare the activities and deliverables to be operational for each session. During each session a debriefing with each group is needed.</p><p>After each session, the teacher controls sites. Mostly complete control of deliverables is essential at each checkpoint. Teacher returns the evaluation at the next meeting during the debriefing. The documents will be annotated and comments can be posted on the site if this feature exists.</p><p>The teacher will keep a notebook of observations to place his/her comments (problems encountered particular difficulties, positive, improvement possible). This book will encourage the evaluation and development of the method implementation.</p></td></tr><tr><td>Work Organization (In presence of the teacher/outside of the classroom /mixed)</td><td>Choose an organizational style for the work:<ul><li>In the presence of the teacher only,</li><li>Outside of the classroom only,</li><li>An hybrid work-style.</li></ul><p>Describe the distribution of tasks during the sessions if you choose an hybrid work</p></td><td>The method and the limited time involve a hybrid organization.<br/><p>Technological Support for enabling and supporting this organization.</p></td></tr><tr><td>ICT Instrumentation</td><td>Study and propose ICT instrumented solutions that could help to deploy the device and foster learner work<br/><p>You can choose to use collaborative platforms, such as Moodle or available technologies, such as Wiki, Blog, e-mail, chats and/or forums.</p><p>Remember to use a variety of tools<ul><li>Editorial</li></ul></p></td><td><p>Communication, logging and filing of documents: Each learner group will deposit this work and the return of its activities through a website or a blog.</p><p>The teacher will note the address of the blog at the first meeting.</p><p>Communication: E-mail of the teacher may be used in case of urgent and if justified (address provided at the first meeting).</p><p>Edition: Using a suite available at the IUT (released documents in pdf format)</p></td></tr></table>
* Collaboration

* Communication

* Development

which can promote learner activities and facilitate your work.

It is important to specify the format of deliverables that you wish to obtain: pdf, doc, jpg or another format.

Development: A Computer Assisted Design tool for modeling and a Database Management System for implementation.

REFERENCES

AFNOR, 1996. Formation professionnelle – Terminologie (professional Training – Terminology), Norme AFNOR X50-750, July.

AKINOĞLU O. AND ÖZKARDEŞ R., 2007. The Effects of Problem-Based Active Learning in Science Education on Learners' Academic Achievement, Attitude and Concept Learning, *Journal of Mathematics, Science & Technology Education* 3(1), 71-81.

ALLY, M. 2008. Foundations of educational theory for online learning. In *The Theory and Practice of Online Learning, second edition*. Terry Anderson Ed, 3-31.

ANDRE, T., 1986. Processes of reading comprehension and the teaching of comprehension. *Historical Foundations of Educational Psychology*. Glover, J., and Ronning, R. R. Ed. Plenum, New York, 259-296.

BAKER, A., NAVARRO E.O. AND VAN DER HOEK, A., 2005. An experimental card game for teaching software engineering processes, *The journal of Systems and Software* 75, 3-16.

BARR R. B. AND TAGG J., 1995. From Teaching to Learning - A New Paradigm for Undergraduate Education, in *Change*, November/December, 13-25.

BASSELLIER, G., REICH, B.H., BENBASAT, I., 2001. Information Technology Competence of Business Managers: A Definition and Research Model, *Journal of Management Information Systems*, Volume 17, Number 4, 159 - 182, ME Sharpe Ed.

BIGGS, J. B., 1999. What the learner does: teaching for enhanced learning. *Higher Education Research and Development*, 18 (1), 55-75.

BLOOM, B., 1968. Learning for mastery. Instruction and curriculum. *Evaluation Comment*, 1(2), 1-5.

BRUNER, J. S., 1960. The act of discovery. *Harvard Educational Review*, 31, 21-32.

BRUNER, J. S., 1966. *Toward a theory of instruction*. Harvard university press. Cambridge.

CALLAHAN, D. AND PEDIGO, B., 2002. Educating experienced IT professionals by addressing industry's needs, *IEEE Software* 19, Sep.-Oct, 57-62.

CHEN, D., STRROUP, W., 1993. General System Theory: Toward a Conceptual Framework for Science and Technology Education for All. *Journal of Science Education and Technology*, Vol. 2, No. 7, 1993

COLLINS, A., BROWN, J.S., NEWMAN, S.E., 1989. *Cognitive apprenticeship: Teaching the craft of reading, writing and mathematics*. In L.B. Resnick (Ed.), *Knowing, learning and instruction: Essays in honor of Robert Glaser*, 453-494. Hillsdale, NJ: Erlbaum.

CONN, R., 2002. Developing Software Engineers at the C-130J Software Factory. *IEEE Software* 19(5), 25-29.

CRAIK, F., LOCKART, R, 1972. Levels of Processing: A Framework for Memory Research. Journal of verbal learning and verbal behavior, 671-684.

DESSUS, P., 2008. Quelles idées sur l'enseignement nous révèlent les modèles d'Instructional Design ? (What ideas about teaching reveal patterns of Instructional Design ?). *Revue Suisse des sciences de l'éducation*, 2006, 28 (1), 137-157.

DEWEY, J., 1916. *Democracy and education*, New York: The Free Press.

DEWEY, J. 1960. *The quest for certainty*. New York: Capricorn.

DICK, W., CAREY, L AND CAREY, J.O. 2011. The systematic Design of Instruction (7th ed.). Allyn & Bacon, 1-12. ISBN 0205412742.

ENGELMANN, 1997. Direct Instruction. *Direct Instruction Reading*. Chapter 22. Prentice Hall. D. Carnine, J. Silbert, E. Kameenui Eds, 371-389.

ERAUT, M., 2000. Non-formal learning and tacit knowledge in professional work, *British Journal of Educational Psychology* 70 (1), 113-136

ERTMER, P., NEWBY, T, 1993. Behaviorism, Cognitivism, Constructivism: Comparing Critical Features from an Instructional Design Perspective

EVELEENS, J.L., VERHOEF, C., 2010. The Rise and Fall of the Chaos Report Figures. *IEEE Software* 27(1), 30-36.

GAGNE, R. M., BRIGGS, L. J., 1974. *Principles of instructional design*. Londres: Holt, Rinehart, and Winston.

GAGNE, R., 1985. The conditions of learning and the theory of instruction (4th ed.), New York: Holt, Rinehart, and Winston.

GIBSON, I., 2001. Group Project Work in Engineering Design Learning Goals and their Assessment, *International Journal of Engineering Education* 17 (3), 261-266.

GIBSON, I., 2005. Designing Projects For Learning, Section 2: Designing Enquiry and Problem-based Learning, In *Handbook of Inquiry and Problem-based Learning Irish Case Studies and International Perspectives*, Barrett T., Mac Labhrainn I. and Fallon H. Eds, Publisher: Centre for Excellence in Learning and Teaching, NUI Galway and All Ireland Society for Higher Education (AISHE), Dublin.

GOODYEAR, P., 2002. Psychological foundations for networked learning. *Networked Learning: Perspectives and Issues*. C. Steeples & C. Jones Eds, London: Springer-Verlag, 49-75.
GLASS, R. L., 2006. The Standish Report: Does It Really Describe a Software Crisis? Communications of the ACM. 49(8), 15-16.

HEWETT, T., BAECKER, R., CARD, S., CAREY, T., GASEN, J., MANTEI, M., PERLMAN, G., STRONG, G. AND VERPLANK, W., 1996. Curricula for Human-Computer Interaction, ACM SIGCHI, available at: http://sigchi.org/cdq/.

HUART, J., KOLSKI, C. AND SAGAR, M., 2004. Evaluation of multimedia applications using inspection methods: the cognitive walktrough case, *Interacting with computers* 16 (2), 183-215.

JACKO, J.A. AND SEARS, A., 2003. *The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications*, Lawrence Erlbaum & Associates.

KENNY, R, ZHANG, Z., SCHIER, R A., 2005. A Review of What Instructional Designers Do: Questions Answered and Questions Not Asked, Canadian Journal of Learning and Technology. Volume 31(1).

KOLSKI, C., 1997. *Interfaces homme-machine, application aux systèmes industriels complexes – 2ème édition (Human-Machine Interfaces, application to complex industrial systems - second édition)*. Éditions Hermès, Paris.

KOLSKI, C., LOSLEVER, P., SAGAR, M., 2012. The performance of future designers on the specification of supervisory HCI: case study of a simulated work situation. *Cognition, Technology & Work* 14 (2), 107-128.

KOLSKI, C., SAGAR, M., LOSLEVER, P., 2004a. Experiment based on participative ergonomics and performed as part of a supervision engineers' training course. In E.F. Fallon, W. Karwowski (Eds.), *Proceedings HAAMAHA'2004 "Human and Organisational Issues in the Digital Enterprise"* (25-27 August), 1, National University of Ireland, Galway, 336-346

KOLSKI, C., SAGAR, M., LOSLEVER, P., 2004b. Spécification d'IHM dans les systèmes critiques : retour d'expérience sur une pratique en enseignement de l'IHM (HMI specification in critical systems : experience in HMI teaching). *IHM 2004, International Conference Proceedings Series, ACM Press, Namur, 157-164, DOI= http://doi.acm.org/10.1145/1148613.1148635*

LABOUR, M., KOLSKI, C., 2010. A pedagogics pattern models of blended E-Learning. A step towards designing sustainable simulation-based learning. in A., Tzanavari & N., Tsapatsoulis (Eds.), 2010, *Affective, Interactive and Cognitive Methods for E-Learning Design: Creating an Optimal Education Experience*, IGI Global, Hershey, PA (USA), 114-137.

LE BOTERF, G., 2002. *Développer la compétence des professionnels (4e édition)* (Developing professionals' competence – 4th Edition), Paris, Éditions d'Organisation.

LEPLAT, J. AND PAILHOUS, J., 1981. L'acquisition des habiletés mentales, la place des techniques (The acquisition of mental skills, the role of technical). *Le travail Humain*, vol. 44, n° 2, pp.275-284.

LETHBRIDGE, T.C., 2000. Priorities for the education and training of software Engineers, *The Journal of Systems and Software* 53, 53-71.

MAHATODY, T., SAGAR, M. AND KOLSKI, C., 2010. State of the Art on the Cognitive Walkthrough method, its variants and evolutions, *International Journal of Human-Computer Interaction*, 26 (8), pp. 741-785.

MAUREL, M., 2009. The explicitation interview: examples and applications, in ten years of viewing from within, *Journal of Consciousness Studies*, 16 (10-12), edited by PETITMENGIN C., 58-89, Exeter EX5 5YX, UK.

MAYES, T., DE FREITAS, S., 2004. Review of e-learning theories, frameworks and models. commissioned review report as part of the JISC-funded e-pedagogy desk study on e-learning models. http://www.jisc.ac.uk/uploaded_documents/Stage 2 Learning Models (Version 1).pdf

MOLENA, M., 2007. The ADDIE Model. In *Educational Technology: A Definition With Commentary*. 2nd Edition. Lawrence Erlbaum Associates Inc. Ed.

MONTMOLLIN, M., 1986. *L'intelligence de la tâche* (The intelligence of the task). Eléments d'ergonomie cognitive, Berne : Peter Lang Edition

NIDIFFER, K. E., 2007. Adressing the Software Engineering Challenges over the Years and into the Future. Software Tech, 10:3, Department of Defense Information Analysis Center, October 2007.

NIELSEN, J. AND MACK, R., 1994. *Usability Inspection Methods*, John Wiley & Sons, New York.

NORMAN, D.A., 1986. *Cognitive engineering*. In D.A. Norman & S.W. Draper (Eds.), User centered systems design: New perspectives in human-computer interaction, Lawrence Erlbaum Associates, Hillsdale, 31-61.

PAQUETTE, G., CREVIER, F. AND AUBIN, C., (1997) Méthode d'ingénierie d'un système d'apprentissage (MISA) (Engineering method of a learning system (MISA)), *Revue Informations In Cognito*, numéro 8, 1997.

PEARS, A.N, DANIELS, M., 2003. Structuring CSEd Research Studies : Connecting the Pieces. *ITiCSE'03*. ACM. June 30-July 2, Thessaloniki, Greece.

PIAGET, J., 1980. The psychogenesis of knowledge and its epistemological significance. *Language and learning*. In M. Piattelli-Palmarini Ed., Cambridge, MA: Harvard University Press.

POLSON, P.G. AND LEWIS, C.H., 1990. Theory-based design for easily learned interfaces, *Human-Computer Interaction* 5 (5), 191-220.

RICHARDS, D. 2009. Designing project-based courses: With a focus on group formation and assessment. *ACM Transactions on Computing Education*, 9, 1, Article 2 (March 2009), 40 pages.

SCHUNK, D. H., 2012. *Learning theories – An educational perspective*. Pearson. 6th Edition. ISBN-13: 978-0-13-707195-1

SHUELL, T.J., 1986. Cognitive conceptions of learning. *Review of Educational Research*, 56, 411-436.

SIMS-KNIGHT, J.E., UPCHURCH, R.L., AND FORTIER, P.A., 2005. Simulation Task to Assess Learners' Design Process Skill. *Proceedings of the 35th Annual Frontiers in Education Conference*. Volume 4, 19-22 October. Indianapolis, F4G-1- F4G-6.

SKINNER, B.F., 1953. *Science and Human Behavior*. New York: Macmillan.

SKINNER, B. F., 1958. Teaching machines. *Science* 128 (967-77), 137-58.

SKINNER, B.F., 1974. *About behaviorism*.New York: Knopf.