# A Framework for Monitoring and Evaluation of Learning Processes

Lucie Pekárková, Patrícia Eibenová, Tomáš Pitner

<figure><img src="image_1.png" /></figure>

## To cite this version:

Lucie Pekárková, Patrícia Eibenová, Tomáš Pitner. A Framework for Monitoring and Evaluation of Learning Processes. 10th International Symposium on Environmental Software Systems (ISESS), Oct 2013, Neusiedl am See, Austria. pp.525-532, 10.1007/978-3-642-41151-9_49. hal-01457484

HAL Id: hal-01457484

https://inria.hal.science/hal-01457484

Submitted on 6 Feb 2017

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A Framework for Monitoring and Evaluation of Learning Processes

Lucie Pekárková, Patrícia Eibenová, and Tomáš Pitner

Masaryk University, Faculty of Informatics
Laboratory of Software Architectures and Information Systems
Botanická 68a, 602 00 Brno, Czech Republic
{pekarkova, eibenova, pitner}@fi.muni.cz

**Abstract.** The paper will describe the features and show advantages of process orientation in all phases of university course design and implementation applied in an introductory course on environmental science at Masaryk University, Faculty of Informatics in Brno, Czech Republic. It will demonstrate the principles of process-oriented learning design and its support in the open-source based Medusy project. Primarily, it will show the integrated approach to learning process design, tooling, runtime environment, and learning process pattern repository.

**Keywords:** process-oriented teaching and learning, Medusy project, course on introduction to environmental science

## 1 Processes in Learning

The contemporary e-learning tools very successfully manage the classic paradigm of teaching, i.e. relatively uni-directional flow of information from teacher to students and subsequent assessment of students. Due to the massive arrival of web technologies and the Web 2.0 phenomena, new modern tools can provide students' feedback, their mutual collaboration or cooperation on content of the lectures (wiki, personal blog, chat etc.). The environmental education is not an exception [10]. The research [7] suggests that well prepared e-learning courses can be even more effective than face-to-face lectures and in most cases reaches the same results as the conventional form of teaching.

The introduction of roles, definition of rules, procedures, repeatable processes and some principles of process management can contribute to a comprehensive learning environment. That also brings advantages such as refactoring and optimization of processes, monitoring and formalization of educational principles and procedures.

Effective use of processes goes along with proper methodology for their management. That is why we focused on principles of process management (Business Process Management – BPM). Even though processes and process management come from business we can find them in non-business domain. In those cases is the process management focused mainly on the technological aspect and the
management part is largely influenced by target domain. That is also a case of
BPM in e-learning. citeleyking

We can say that the learning itself is a process. The owner of the process is mainly a teacher. The main (learning) processes are created by activities and events that form the content or goal of the lecture. In connection with time they create the learning plan. Finding such learning processes is a subject of empirical research (or a pedagogical problem). The support processes participate on creation and management of the learning environment and as well as in the classical case provide the additional functionality for main processes.

In the business world, the BPM implementation leads to one goal – cost reduction and profit improvement. In the e-learning domain the goal could be higher efficiency and quality of education. Implementation and automation of processes in e-learning should project as a combination of direct (students have clearly defined tasks and duties including deadlines; easier information sharing and a knowledge base; opportunity to use sophisticated services and utilities, especially for technology-oriented courses) and non-direct (reducing number of administrative tasks, their automation and teachers’ team cooperation support leads to even distribution of responsibilities and competences and the teacher has more time to prepare the content; instant overview of the situation status, completed and uncompleted tasks; possibility of continuous evaluation of the effectiveness and further learning process improvement) effects.

## 2 Process Modeling

As we agreed, creating the learning processes is largely a pedagogical problem - every course or lecture requires different approach. The use of the educational patterns can in many ways simplify creation and optimization of learning processes and at the same time improve the quality of learning. Formalization of each pattern via processes could help during the subsequent discussion or repeated use. [3] In our perception the educational pattern is a small reusable sequence of steps that are included in one blended learning activity. Patterns should be used as building blocks for more complex blended courses. [1] A pattern should define general structure of the activity and should be customizable. One of our aims is to build community based repository of such patterns. Patterns should be built on best practices in blended courses and each pattern should define standard structure of activity that can be extended according to specific needs of other (re-)users. In context of learning activities it should serve as a guideline, how to use certain educational pattern.

BPMN 2.0 was chosen because this modeling language has to be capable of
immediate execution after modeling in contrast to BPMN 1.1.

## 3 Medusy Project

The Medusy project is in an early phase of development such as analysis and
design. We have done one functional prototype of process based e-learning appli-
cation, tool for learning process modeling, pattern repository and now we are in
next iteration where we try to integrate monitoring and evaluation tools into the
project. A prototype of the engine was designed which enabled to analyze pos-
sibilities of process management and patterns utilization in e-learning/blended
learning.

**3.1 Process Engine**

The main technology used in the process engine is *Activity BPM* based on BPMN 2.0. It is an evolution of *jBPM* – an environment working on the *Process Virtual Machine* (PVM) principle. The PVM is an architecture design pattern for process engines. The structure of the process definition is separated from the implementation of the different activity types. And the runtime data structure for process executions is based on a pointer to an activity in the process graph. Activity types and hence whole process languages become pluggable on that single core engine. Main characteristics of this environment are stability, high performance, easy customization, transaction support and asynchronous task support. The resulting architecture is directly derived from used technologies. The main principle is to access the Activity runtime environment from within *Ruby on Rails* (RoR) application.

**3.2 Process Modeling**

For better user experience an open-source modeling tool was created based on
the *Oryx* editor. The Oryx editor was enhanced with modeling of learning pat-
terns for advanced users that are familiar with BPMN 2.0 and with modeling of
learning processes in simple notation by parametrizing the pre-modeled patters
for casual users. After the learning process is modeled, it can be exported to
XML format that is executable in the Activity engine prototype.

4 Learning Patterns

The concept of using patterns was for the first time established in architecture.
Christopher Alexander captured and described applied expert design advices
as architectural design paradigms. Similar principles were used several years
later in the software engineering for describing distinguish and frequent design
issues in object-oriented software systems. Years later, pedagogical domain was
explored as well. In this context, patterns are used to capture best practices
from the educational area. It is crucial that patterns are able to solve one issue
appearing in various contexts at a time. [2]Another more adjusted perspective is
not searching for best practices, however to locate more various solutions. Final
selection of suitable pattern is supposed to be a teachers preference. [3]
## 4.1 Patterns for an Environmental Course

Within the project, there are two environmental-related courses taught at Faculty of informatics that were innovated by adding new e-learning and interactive elements. Enhanced process models of the courses were created. Innovation of environmental courses as well as the whole project stands on a principle of using *learning patterns*. Creating a course model tends to be as an illustration how the models can be created in the future. In this paper, we present a model of the Environmentalistics course, taught by one of the authors at Faculty of informatics, Masaryk University Brno.

## 4.2 The Course on Environmentalistics

*Environmentalistics* is a basic environmental course taught at Masaryk University, Faculty of informatics. The course is an introduction to the topic of environmental science with focus at global environmental problems.

The education is organized into interactive lectures, followed by a colloquial exam. During the course, two essays are assigned. This is very common pattern in education – lectures, homework and an oral colloquial exam.

Innovation requirements of this course should fit into these set rules, therefore *no increased demands* on students or teacher are required. Suitable option is to integrate e-learning elements into the course, thus create a blended-learning course. An advantage of this concept is that students can prepare before coming to lectures and devote more time to extended topics than repeating basic facts. Lectures demand active participation. The Environmental information systems course extends this basic course, see [5]. It is devoted to managing, monitoring a evaluating of data in environmental information systems and likewise design of these very systems.

## 4.3 Principles of Modeling Learning Processes

In innovation of both courses, learning patterns were used through current pattern database. Due to this method, any teacher would enhance own courses by using a tool which can provide required patterns – educational objects. Educators are expected to make their choice and compile a whole course.

The syllabus of the Environmentalistics were described in informal specification – how is the course organized and running, conditions of evaluation and so forth. The very first step, just before the very modeling is creation of a structured list. This list has a clear start and termination and every further step in the list is a part of a process. According to complexity of steps on the list, it is possible to moderate an abstraction scale of the modeling tool. For instance, a user insists on placing a team project into the course, however he does not have accurate conception. There exist several options how to organize a team project in a course and the user does not have to be familiar with all of them. Therefore there is a possibility of choosing present patterns from the repository. Eventually the whole course is created by compiling single elements together.
Despite of this light description from the users point of view, there is a BPMN model behind every pattern. Compiled courses have to be functional and make sense likewise from the technical point of view, which is not easily controllable. Therefore automatization is only partial and the created model should be manually validated.

## 4.4 Demonstration of Enhanced Formal Course Model

Final compilation of the complex course model is a pedagogical matter. A teacher decides which learning object should take place in the final course and how to connect them. In this paper, we present the enhanced concept of final colloquium. Colloquium is a method of examination, where small group of student discuss learned knowledge with the teacher. Each student also defends elaborated essay.

The concept of final colloquial exam is slightly enhanced in a new model, see Fig. 1. Students are divided into groups of three. Each student is supposed to elaborate his/her own final essay, submit it, afterwards read essays of two colleagues and write a report which will be discussed.

<figure><img src="image_3.png" /><figcaption>Fig. 1. "Colloquial exam" process</figcaption></figure>

Model of a colloquium is a standard BPMN 2.0 diagram and consists mostly of collapsed subprocesses. Each subprocess represents a set of activities within a business process. A subprocess contains another whole business process with its start and termination and sequence of tasks or another subprocesses. For instance, Fig. 2 shows that subprocess *Publish topics* contains another process.

For every learning object in the course, there is a business model. Particular subprocesses have a precise order and together they compose a whole course.
<figure><img src="image_4.png" /><figcaption>Fig. 2. "Publish topics" subprocess</figcaption></figure>

# 5 Pattern repository

The next step within the Medusy project is creation of a *learning patterns repository*. The repository is a module in the framework maintaining a database of the most used patterns in the educational domain. By module we mean a standalone application connectable to the other part of the system such as modeling tool and execution engine.

The proposed repository deals with several other challenges that define this solution. It is possible to achieve proposed attributes by suitable merge of current modules used in the project. At the moment, the database is deeply vast and unorganized, however forms a core component to the pattern repository. Demands of the proposed repository are as follows:

- interconnection with the modeling tool and execution engine;

- structuring patterns according to function, popularity, subject domain or the scale of abstraction;

- simple and friendly user interface;

- adjustment of the pattern by using the BPMN;

- documentation and precise pattern description;

- validation of formal correctness of a pattern.

## 5.1 Codification vs. Personalization Approach

Another important matter is a scale of customization and whether to provide maximal scale with the possibility of creating custom models or offer fixed steady patterns, see [4].

The codification approach is focused on using the patterns repeatedly – use one single pattern over and over again is the main goal. In the beginning, there is a concern on creating an exquisite database of patterns. User chooses a pattern and sticks with the solution that should fit perfectly. There is a minimal need
for optimization considering the high input at the beginning. By contrast the personalization approach enables representing the model thoroughly. The personalization encourages innovation, however, it exceeds our pattern definition, see [9].

## 5.2 Optimization of the Modeling Tool for Users

We also assume two kinds of users, an inexperienced one, who has not any experiences with modeling in BPMN 2.0 notation. This user has an opportunity to use predefined patterns and assemble them into a new course. Another option is to choose from predefined whole courses assuming there is no need for added adjustments. Second kind of user, an experienced one, has already some knowledge about BPMN 2.0 notation and knows how to model simple processes. In this case, it is possible to create entirely new course according to special demands. This option enables maximal scale of customization. However, the preferable attitude dedicated to the majority of users, is submitting prepared patterns.

## 5.3 Integration of the Repository into the Framework

Besides the current database of patterns as an essence of the proposed pattern repository, there are other modules in the framework. The repository will be a core component to the modeling tool. A teacher creates a model of a course with the aid of the modeling tool and simultaneously uses existing patterns from the repository. The current modeling tool is based on Oryx editor so it is essential that the repository will inherit applied technologies and solutions. A prepared and validated model of a course is an executable model as well. Models are executed in the functional prototype of process engine. There are no compatible issues due to the fact we use BPMN 2.0. Executable XML file is a requirement for execution of the model in the process engine.

# 6 Conclusion and Future Work

The project Medusy tries to leverage best practices in person-centered technology-enhanced learning. The project introduces a process-oriented approach to course design, development and education flow in general. The designed framework provides background for modeling, execution, monitoring and evaluation of education processes in academic and commercial environment. Process models of two environmental courses at Masaryk University, Faculty of Informatics have been created. The models are described in up-to-date BPMN 2.0 notation and executable in the process engine. All the models were created in systems modeling tool and compiled from the learning patterns from the project repository. Obtained data and findings will be analyzed and the output will be used in the upcoming module for monitoring and evaluation of educational processes.
**Acknowledgement.** This work was supported by the European Unions territorial cooperation program European Territorial Co-Operation Austria-Czech Republic 2007-2013 of the under the EFRE grant M00171, project iCom (*Constructive International Communication in the Context of ICT*).

## References

1. Avgeriou, P., Papasalouros, A., Retalis, S., Skordalakis, M.: Towards a pattern language for learning management systems. In: Educational Technology & Society, 6(2), pp. 11-24 (2003)

2. Bergin, J. et al (Eds.) Pedagogical Patterns: Advice for Educators. Joseph Bergin Software Tools. http://www.pedagogicalpatterns.org (2012)

3. Derndl, M., Motschnig-Pitrik, R.: The role of structure, patterns, and people in blended learning. In: The Internet and Higher Education, 8(2), pp. 111-130 (2005)

4. Hansen, M.T., Nohria, N., Tierney, T.: What's your Strategy for Managing Knowledge? In: Harvard Business Review, March/April 1999, pp. 106-116 (1999)

5. Hřebíček, J., Kubásek, M.: Environmentální informační systémy (Environmental information systems). CERM, Brno, http://www.iba.muni.cz/res/file/ucebnice/hrebicek-environmentalni-informacni-systemy.pdf (2011)

6. Leyking, K., Chikova, P., Loos, P.: Competency and Process-Driven e-Learning – a Model-Based Approach. In The Electronic Journal of e-Learning, 5(3), pp. 183–194 (2007)

7. Means, B. et al.: Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies, U.S. Department of Education. http://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf (2010)

8. Ministr, J., Stevko, M.: Human Resources Requirements for Professional Management of ITSCM Process. In: Doucek, P and Chroust, G and Oskrdal, V: IDIMT-2010: INFORMATION TECHNOLOGY – HUMAN VALUES, INNOVATION AND ECONOMY. 18th Interdisciplinary Information Management Talks, Jindrichuv Hradec, CZECH REPUBLIC, SEP 08-10, 2010, Schriftenreihe Informatik, Vol 32. Springer, Berlin Heidelberg (2010)

9. Reimer, U., Heck, U., Streit, S.: Collaboration-Oriented Knowledge Management Using Interaction Patterns. In: Practical Aspects of Knowledge Management, Lecture Notes in Computer Science, Volume 5345, 2008, pp 26–37. Springer, Berlin Heidelberg (2008)

10. Talas, J., Gregar, T., Pitner, T.: Semantic Wiki in Environmental Project Management. In: Hrebicek, J; Schimak, G; Denzer, R. 9th IFIP WG 5 11 International Symposium on Environmental Software Systems (ISESS 2011), Brno, CZ. IFIP Advances in Information and Communication Technology, Vol. 359, pp. 437–444 (2011)