# E-Learning Maturity Model – Process-Oriented Assessment and Improvement of e-Learning in a Finnish University of Applied Sciences

Ilkka Haukijärvi

► To cite this version:

Ilkka Haukijärvi. E-Learning Maturity Model – Process-Oriented Assessment and Improvement of e-Learning in a Finnish University of Applied Sciences. IFIP Conference on Information Technology in Educational Management (ITEM) and IFIP Conference on Key Competencies for Educating ICT Professionals (KCICTP), Jul 2014, Potsdam, Germany. pp.76-93, 10.1007/978-3-662-45770-2_9.hal-01342688

HAL Id: hal-01342688

https://inria.hal.science/hal-01342688

Submitted on 6 Jul 2016

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
E-learning Maturity Model – Process-oriented
Assessment and Improvement of e-Learning in a Finnish
University of Applied Sciences

Ilkka Haukijärvi

Development Manager, Tampere University of Applied Sciences, Finland
ilkka.haukijarvi@tamk.fi

**Abstract.** In order to institutionalize e-learning within an institution its significance as an object of continuous process assessment, process improvement and process management must be acknowledged. One of modern higher education institution´s key assets is its capability to develop and sustain social and physical structures and capabilities which enable the development and execution of high quality e-learning. In this article, a case study is presented, during which e-learning related process assessment and evaluation of eLearning Maturity Model (eMM) was carried out in a large Finnish university of applied sciences. The evaluation of the eMM was committed as part of a comprehensive process assessment of the case institution. The eLearning Maturity Model was considered beneficial, although some criticism appeared. It provided a structured, although quite a resource heavy, approach to form a comprehensive and multilevel overview of the institution´s current status of processes surrounding the e-learning. By utilizing the eMM, a more informative basis for future development of processes was formed. Based on the process assessment, the case institution´s capabilities on higher dimensions of capability are generally lower than in lower dimensions. This potentially results in high variation of performance and quality within e-learning.

**Keywords:** E-Learning Maturity Model, eMM, e-learning maturity, process maturity, process management

# 1 Introduction

An institution (organization) is dependent on its capabilities and knowledge base in
terms of making strategically right decisions. Institutions – whether private or public
– have faced an age of digitalization of learning and training. The ‘E’ has become an
essential part of different domains within public and private sector institutions.
Ruohonen & Multisilta [1] write that e-business, e-work and e-learning form an
integrated entirety, which is to be guided and managed through strategic perspectives.
In addition, when developing and deploying e-learning, it is crucial to understand the
pedagogic difference compared to traditional learning. In many case, pedagogic
approach that supported face-to-face lecturing is transferred to e-learning environment
as such. This approach will lead to failure most likely. Strategic development of e-
learning should aim for more effective learning of individuals, teams and institutions.
Organizational learning is dependent on the level of integration between individual, group, organization and inter-organizational learning processes. Ruohonen & Multisilta [1] state that organizational learning needs to be addressed and a lot of work needs to be done in order to integrate processes for learning in different domains.

Higher education institutions are facing pressures to develop the effectiveness and quality of their operations. The accountability of higher education institutions, also in terms of performance and quality, has seen a significant increase during the last two decades [2]. The development has led to greater utilization of various metrics and process improvement activities on a wide scale. In Finland for example, every university of applied sciences is accountable to the Ministry of Education, and multiple different metrics, mostly quantitative, are being used to determine the performance and further, the annual funding. The mandate and funding of an applied sciences institution is largely based on these metrics and reports. From internal quality assurance and development point of view, both quantitative and qualitative metrics and assessment practices are being utilized in forming a better understanding of how well an institution performs in a given area of assessment. The goals of assessment and evaluation are to form a solid and informative basis for institutional and individual development. The development initiatives and activities are carried out in order to meet the expectations and mutual agreements with key stakeholders. An organisation needs to be managed and developed from strategic perspectives and based on collected information and careful analytics. Explicit objectives need to be set and communicated throughout the organization. Planning, development, assessment, execution and management benefit from structured and rigor approaches, which provide guidance and support for the continuous improvement, but also provide means to enhance the knowledge base of an organisation. A learning organization creates and sustains circumstances that support collaborative construction, open sharing, storing and utilization of knowledge [3].

The role of e-learning has grown essentially during the last two decades. Digitalization of education is considered as a major reforming megatrend in higher education. Many higher education institutions are engaged in studying and utilizing various e-learning solutions. As the role of e-learning is essential part of a modern day higher education institution, assessing the effectiveness of e-learning in a comprehensive manner has become crucial. Institutional assessment and development has a crucial role in ensuring and enhancing the circumstances that support the achievement of desired outcomes. In addition to emphasizing continuous development of e-learning environments and technologies, learning artefacts, learning processes and pedagogies, there should be a complementary emphasis on evaluating institutional processes affecting and defining the circumstances in which the e-learning is developed, deployed and supported.

In this article, a case study is represented, during which an assessment of process maturity was executed in large Finnish University of Applied Sciences. The E-learning Maturity Model CORE version 2.3 by Marshal (2006) was utilized in the assessment. The goal of the study was to form an overview of the case institution's current state of e-learning process maturity and capability and to collect evidence of the utility of the eMM as qualitative process assessment framework in the local context.
## 2 Business Process Management – A theoretical and practical Framework for Process Development

Within rapidly changing environment, institutions must have capabilities for continuous development. Pressures for change occur from within and from outside the organization. In this kind of environment, the development and assessment of organization's processes and its capability to dynamically reform the resource base play fundamental role in maintaining and developing functionality and competitiveness. Helfat et al. [4] define dynamic capabilities as organization's ability to create, expand or reform its resource base appropriately – tangible assets, intangible assets, personnel resources and all the capabilities that it owns or has an entitled access to.

The quality of an organization's processes correlate to the effectiveness of the organization. Business Process Management (BPM) is widely applied and established model for and approach to development of organizations' effectiveness and competitiveness. The theoretical foundation on which the BPM has been developed on, is the work on statistical process control by Walter Shewhart and W. Edwards Deming and Business Process Re-engineering (BPR) by Michael Hammer [5].

Hammer [5] criticized that the statistical quality approach emphasized too heavily on small scale processes or even work activities, rather than on processes that have real strategic effect in terms of value proposition of the enterprise. Another restriction of the quality approach was heavy focus on eliminating variations and ensuring constant quality within processes or sequences of work [5]. Hammer [5] saw this as restriction since constant quality does not necessarily lead to fulfilling the value proposition – the customer expectations. The absence of greater emphasis on continuous optimization and pursuing for optimal circumstances and fulfilment of the expectations of key stakeholders was seen problematic by Hammer. Hammer's own work with Business Process Re-engineering has had its impact on the BPM as well. In BPR, processes were defined as end-to-end processes that function throughout the organization and produce value for the customers [5]. End-to-end processes are processes that are crucial and meaningful in terms of the organization's competitiveness [5]. In the quality approach by Deming and Shewhart the development of processes was continuous activity that based heavily on quantitative metrics, whereas in BPR the development was staged and no clear and defined approach for using pre-defined metrics was introduced. Process planning and design had a large role in BPR, whereas the quality approach relied heavily in assessing the outcomes.

During the last decade, Business Process Management was formed on the basis of quality approach and BPR, introducing a modern approach to the management of processes [5]. In BPM, the development is focused on large scale processes that have essential role in realization of value proposition – defining, designing, assessing and developing these processes as continuous cycle.
## 2.1 Cycle of Process Management

Careful analysis of systematically collected information on the current state is the basis for development. Information analysis forms the evidence base for planning of development. After planning the development, the changes are realized and effects of the changes are assessed and measured. After these stages, new processes are standardized and implemented throughout the organization [6]. Through careful planning and definition of processes, the probability of achieving higher level process performance grows. In order to manage a massive network of processes, which rely heavily on both physical and social assets, a structured approach to development can be beneficial. Mutafelija & Stromberg [6] state that a structural approach supports planning, guides development and provides a clear framework for assessing development efforts and outcomes.

One example of a structured qualitative process assessment and development framework is the Capability Maturity Model Integrated (CMMI). Key idea of maturity models in general, is that organizations need to develop their capabilities within processes in order to achieve higher level maturity of the processes. This kind of approach provides a comprehensive and holistic approach to assessment and development of processes. Kasse [7] writes that CMMI includes a group of plan, do, check and act –activities which are known to be essential phases in the famous process development PDCA-cycle. CMMI’s purpose is to support continuous process development by providing a structured framework for the analysis of current state, approach to and guidelines for development itself and assessment of development.

Process management cycle of BPM (PMC) has a lot common with the PDCA cycle. The PDCA – plan, do, check, act – is widely recognized and applied process development cycle. The idea is to compare current state of processes with the optimal state and goals. Process management cycle includes multiple different phases as described in Figure 1.

<figure><img src="image_2.png" /><figcaption>Fig. 1: Process Management cycle as described by Hammer [5].</figcaption></figure>
Defined goals – whether quantitative, qualitative or both – must reflect directly to the planning of process development. Otherwise there is a risk of misdirected development: conflict between development efforts and organizational goals occurs. Thus, it is vital to understand that in the assessment of processes and planning of development, the information of the processes' capability itself is not enough. Organization's strategic goals should guide all development.

Failures within processes can occur in two dimensions: processes' design or
processes' execution. If process is continuously underperforming, most likely the
problem lies within design, whereas problems with execution are usually linked to
lack of capabilities and insufficient tools. Roots of the problems within execution
dimension can be hard to recognize and repair. [5].

Institutionalization of processes is key factor in succeeding at process
development. Kasse [6] defines institutionalizing as creation, assessment and
deployment of effective, repeatable and long-lasting processes throughout
organization. To further specify the definition, Kasse adds that when institutionalizing
processes, organization develops practices that provide necessary infrastructural
support and enable organization's continuous learning and evolvement. Lack of
institutionalizing can occur in various ways: processes are managed and executed
inconsistently, processes are vulnerable to changes in human resource base, process
development is not linked to business goals, cost evaluations have no valid history
base, lack of commitment to allocation of resources or infrastructure to development
[7]. Previously mentioned CMMI includes a certain set of goals – generic goals –
which describe the required institutionalizing in order to achieve certain maturity or
capability level [6].

3 E-learning Maturity Model – Description of Key Concepts

Managing and evaluating an institution's e-learning process maturity and capability demands utilization of systematically collected and analysed information to support and guide further development of the area. From strategic management and planning point of view, this information is critical in order to form a relevant and valid basis to decide about long-term resource allocations and other means of support. According to Marshal [8], e-learning process maturity assessment is a complex area alone, and must involve gathering of information that helps to form broader overview to support strategic planning and development of e-learning, rather than focusing too much into small details.

For this purpose was introduced the first version of the E-Learning Maturity Model by Marshal & Mitchel in 2003. In 2006, the second version of the model was published. The E-Learning Maturity Model (eMM) is a structured quality improvement framework which approaches development through assessment of process maturity and capability [8]. The eMM is based on Capability Maturity Model (CMM) and Software Process Improvement Determination (SPICE) model [8]. The eMM is built on a concept that institutions ability to be effective is dependent on its’
capability to sustainably develop, deploy and support high quality processes throughout institution [8].

In the eMM, there are multiple defined processes within total of five defined process categories (areas) (Table 1). The assessment of process maturity and capability is executed through evaluation of a group of practices on five different dimensions of capability. This kind of structure helps approaching wide and complex institutional system holistically with clear vision to the entirety, its' components and interdependency between them. As Marshal [8] states, eMM's advantage is that with it a wide and complex area is broken down into categories that can be assessed separately. Thus, an institution does not necessarily have to utilize whole eMM framework when engaging in assessment. Resource allocations can be directed to more narrowed area within the eMM – for example, to a certain process category.

## 3.1 Process categories

There are total of five process categories: Learning, Development, Support, Evaluation and Organization. Each category includes multiple processes. See the definition of each category in Table 1.

<table><caption>Table 1: Process categories of the eMM [8]</caption><thead><tr><td>Category</td><td>Definition</td></tr></thead><tbody><tr><td>Learning (L)</td><td>Processes that directly impact on pedagogical aspects of e-learning</td></tr><tr><td>Development (D)</td><td>Processes surrounding the creation and maintenance of e-learning resources</td></tr><tr><td>Support (S)</td><td>Processes surrounding the support and operational management of e-learning</td></tr><tr><td>Evaluation (E)</td><td>Processes surrounding the evaluation and quality control of e-learning through its entire lifecycle</td></tr><tr><td>Organisation (O)</td><td>Processes associated with institutional planning and management</td></tr></tbody></table>

Processes within the Learning category are executed in order to maintain and protect circumstances that enable effective and high quality learning, despite of the characteristics of technologies or pedagogies applied [8].

Processes within the Development category are executed to ensure effective and productive utilization of resources. Resources are allocated to development of e-learning infrastructure, materials and course production. In addition, the knowledge base formed during history of development initiatives, such as projects and programs, is utilized to the advantage of current and future e-learning development. Processes within Support -category are executed to secure sufficient technical and pedagogical support and sharing of supportive information to use of students and teachers. Processes within the Evaluation-category are executed to provide comprehensive information basis for planning and execution of e-learning strategy and sustainable development of infrastructure and personnel. Processes within the Organisation-
category are executed to ensure that development of e-learning technologies and pedagogies are planned based on strategic and operational goals [8]. All development must be linked to institution's long-term strategic and operational goals – to avoid controversy between strategy based needs and e-learning development. These processes are focused on strategic, administrative and organizational areas of e-learning which set the preconditions to transition from traditional learning towards e-learning [8].

## 3.2 Practices and dimensions of capability

As with the CMMI, briefly referred to earlier in this article, the eMM enables continuous improvement and assessment of quality with defined practices. In addition, dimensions of capabilities provide the holistic view on assessment and development. Capability's core essence within the eMM is institution's capability to develop to satisfy the needs of students, personnel and whole organization, and maintain support despite of changes with demand and personnel [8].

Capability within five dimensions is defined through assessment of defined practices under each process. Practices are distributed into specific dimensions and grouped into essential core practices and useful practices within each dimension [2]. The dimensions of capability are: *Delivery (1)*, *Planning (2)*, *Definition (3)*, *Management (4)* and *Optimization (5)*. The eMM provides pre-defined practice statements to help assessment of practice specific capability. Through assessing practices is determined the level of maturity of a process, and eventually, the maturity of a process category.

All the dimensions are interconnected. High capability on higher dimension does not ensure high quality process outcomes if capability on lower dimensions – such as delivery and planning – is low. To further describe the interconnectedness, low capability on higher dimensions causes ad-hoc functionality, unsustainability and unresponsiveness. [8]. There is causality between low level capability on higher dimensions and institution's inability to proactively manage and continuously improve itself. Assessing merely process outcomes is questionable, as stated in BPM, since there is a complex and interconnected chain between outcomes and phases leading to outcomes. Thus, a holistic approach is needed for developing the whole chain.

Delivery dimension focuses assessment on process's capability to output desired results on wide scale in an institution. On planning dimension, assessment focuses on how explicit and communicated goals and plans affect and guide processes. Definition dimension is closely related to planning dimension, expanding assessment to more official information guidance, such as standards, formal guidelines and policies which have an effect on process implementation [8]. On management dimensions, assessment is focused on process management and quality assurance. Emphasis on control, measurement and personnel capability is evident. Optimization dimension is dependent on all lower dimensions, meaning that improving capability on this dimension will actualize through improving capability on lower dimensions. Systematic collection and analysis of various types of information and formal approaches to support institutional development and decision making are the
emphasis of assessment. As Marshal [8] states, optimization dimension describes how well an institution has adopted culture of continuous improvement.

## 4 The Study

The original case-study was carried out in 2013. The case institution is among the largest Universities of Applied Sciences in Finland. More detailed institutional information in Table 2. The amount of fully online courses is low compared to the size of the institution. In the year 2013, there were approximately 50 courses which were fully online. From this statistic perspective, the performance was quite low compared to many other similar scale universities of applied sciences in Finland. There are various motivators which endorse transformation towards higher performance and volume in e-learning. Quantitative agreements with the Ministry of Education promote the creation of circumstances that support effective completion of studies. E-learning is one area that has the potential to support more dynamic and effective completion and faster graduation. The results from recent internal inquiries show that there is a demand for higher volume of online courses among the students. In addition, variation of quality between courses has been acknowledged and criticized by the students.

<table><caption>**Table 2:** Case-institutions describing figures in 2012.</caption><tr><td>Total budget</td><td>~72 million euros</td></tr><tr><td>Amount of teaching personnel</td><td>414</td></tr><tr><td>Amount of R&D&I personnel</td><td>34</td></tr><tr><td>Amount of administration personnel</td><td>312</td></tr><tr><td>Total amount of personnel</td><td>760</td></tr><tr><td>The amount of youth bachelor's degree students</td><td>8209</td></tr><tr><td>The amount of adult bachelor's degree students</td><td>1052</td></tr><tr><td>The amount of master's degree students</td><td>453</td></tr><tr><td>The amount of teacher education students</td><td>699</td></tr><tr><td>The amount of specialization studies students</td><td>363</td></tr><tr><td>Total amount of students in degree awarding studies</td><td>9714</td></tr><tr><td>Total amount of students in all studies</td><td>10776</td></tr><tr><td>Amount of fields of education</td><td>7</td></tr><tr><td>Amount of bachelor's degree programmes</td><td>47</td></tr><tr><td>Amount of master's degree programmes</td><td>11</td></tr><tr><td>Amount bachelor's and master's degrees</td><td>1859</td></tr><tr><td colspan="2">Largest fields of education</td></tr><tr><td colspan="2">1. Technology, communication, transport</td></tr><tr><td colspan="2">2. Social services, health and sports</td></tr><tr><td colspan="2">3. Social sciences, business and administration</td></tr></table>

Before this study, it was commonly known by experts that the variation of e-learning quality and is very high within the institution. Some educators are
experienced with e-learning and provide high quality e-learning. On the other hand, there is great majority that does not utilize the potential of e-learning. With this detail in mind, it is important to note that the results of this study are an overview of the institution, not explicating a single unit's – such as a degree programme – results. It is safe to assume that some degree programme's results would stand out notably higher than the overview covered in this study.

For the assessment of the processes, E-Learning Maturity Model CORE version 2.3 was utilized, thus focusing the assessment on essential practices within each process. The aim was to form an overview of case institution's process capability and maturity, and based on the results, to form an informative basis for future development. In addition, additional purpose of the study was to evaluate the eMM itself. Evaluation of such frameworks is an essential part of the development. Knowledge of different kind of structured approaches to development forms an essential part of the knowledge base related to development.

The assessment of processes and the evaluation of eMM were conducted in small workshops. Various domains of the institution were represented in the workshops: the strategic management, pedagogic development, quality assurance, e-learning technology, e-learning pedagogy and operational management. The researcher's professional role in the institution was the leader of e-learning support services.

**4.1 Evaluation of E-Learning Maturity Model – Summary**

In Design Science in Information Systems Research -paradigm (DSISR), understanding and awareness of current issues and correspondent solutions develops during development and utilization of artefacts [9]. The evaluation of eMM in this study is best described as observational evaluation. The evaluation was conducted during the utilization of the artefact, in real higher education environment, for actual purposes. In observational case study, the artefact is studied and analysed in actual business environment [9].

Thorough documentation, systematic monitoring and reporting were partially questioned by the participants. The criticism on formality and control occurred more in discussions with experts who did not work in the field of management and quality assurance. Confrontation between the freedom of expertise and policy based control was evident in some discussions. On the other hand, the majority of formal and systematic aspects were considered very beneficial, for example those addressed to collection and utilization of information to support decision making and development in e-learning. Also, eMM's dimensional approach to assessment helped participants of different backgrounds to contribute effectively to the study. The participants understood the purpose and aim of eMM's formal, structured and comprehensive approach to assessment. The framework guided experts to analyze their institution's processes from different perspectives than they were accustomed to, encouraging – if not forcing – active collaboration of experts from different domains of the institution.

After initial criticism and by developing better understanding of the model, some negative characteristics were eventually considered beneficial. Some informal comparison between other models and eMM, in terms of assessing the state of an institution, was also brought up, even though it was not expected at this stage. In the
light of these single coincidental comparisons, eMM was considered a very thorough model that provides tools for addressing issues explicitly in a complex system. It was not necessarily seen as a replacement but rather as a complementary tool. As a maturity model, the eMM also provides a way to measure and monitor development, which was seen as a positive characteristic, although that aspect could not be actualized in the study. The aim was to form an overview of current state and define basis for future development.

<figure><img src="image_3.png" /><figcaption>Fig. 2: Information Systems Research Framework by Hevner et al. [9]</figcaption></figure>

During the process of evaluating eMM and assessing the institution's processes, better understanding was developed on how a structured approach can be beneficial in assessing the current state when it comes to the capability to consistently develop, deploy and support circumstances for high quality e-learning throughout an institution. Consistency and sustainability is supported by strategies, policies, principles, stake holder involvement, formal and systematic top-down and bottom-up quality assurance and enhancement practices. As a disclosure from the discussions, the model was considered beneficial, although very resource demanding to utilize. Further utilization of eMM as a process assessment tool within the institution is expected.

## 5 Process assessment – Summary

With the exception of Evaluation category, and partially Development category, lower dimension capabilities are generally higher. On higher dimensions, the lack of systematic collection, sharing and utilization of e-learning specific information in
decision making and development is the major factor leading to lower capability. Also lack of formal and defined procedures and policies leads to high variation of process outcomes and therefore to low maturity in many processes. The quotes below do not represent the entire data collected, thus being a narrowed representation of how the maturity was determined. The ratings of the maturity levels are: 4 = fully adequate, 3 = largely adequate, 2 = partially adequate, 1 = not adequate.

Learning category's strength lies in formal and explicit communication and documentation of learning objectives in curricula and course documentations throughout the institution. Although, there is some variation of quality and amount of course related instructions, such as schedules and communication policies. Also, dedicated modules for information literacy and research skills development is a common practice in the case institution. Interaction between students and teachers is enabled by various communication channels, although response times or the usage of the channels are not monitored systematically throughout the institution.

"Clear definitions of desired learning outcomes in curriculum and course documentation. Student learning outcomes and quality is monitored and assessed regularly and this information guides further development of courses."

"There can be large variation of quality of the course related information provided in the actual course environment in the learning management system: Some courses include clearly defined instructions on schedules, response times and preferred communication channels, and are aligned with the information provided in the curricula."

"No monitoring of response times in all domains, but clearly defined response times in ICT-support. The monitoring of the use of different communication channels is mostly occasional and informal."

"Assessment practices vary. Institutional policies define the overall type and quality of feedback provided to students. Teachers are provided support on feedback, but this is mostly occasional and not included in common personnel training."

"Feedback given to students is monitored to some degree. Information on student satisfaction on feedback given to them is informally utilized."

"Information literacy skills development is an implicit part of education throughout the studies, and specific training is provided by the library services. Research skills development is enabled with specific course modules built for the specific purpose and through R&D projects which students participate in."

"The information related to the utility and effectiveness of available information sources and tools is utilized in the planning and development of e-learning."
<table><caption>Table 3: Maturity matrix of process category Learning</caption><thead><tr><th colspan="2">Learning: Processes that directly impact on pedagogical aspects of e-learning</th><th>D1</th><th>D2</th><th>D3</th><th>D4</th><th>D5</th></tr></thead><tbody><tr><td>L1</td><td>Learning objectives guide the design and implementation of courses</td><td>4</td><td>3</td><td>4</td><td>3</td><td>4</td></tr><tr><td>L2</td><td>Students are provided with mechanisms for interaction with teaching staff and other students</td><td>4</td><td>2</td><td>1</td><td>2</td><td>2</td></tr><tr><td>L3</td><td>Students are provided with e-learning skill development</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td></tr><tr><td>L4</td><td>Students are provided with expected staff response times to student communications</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td></tr><tr><td>L5</td><td>Students receive feedback on their performance within courses</td><td>3</td><td>2</td><td>2</td><td>2</td><td>2</td></tr><tr><td>L6</td><td>Students are provided with support in developing research and information literacy skills</td><td>3</td><td>3</td><td>4</td><td>3</td><td>2</td></tr><tr><td>L7</td><td>Learning designs and activities actively engage students</td><td>3</td><td>3</td><td>3</td><td>2</td><td>2</td></tr><tr><td>L8</td><td>Assessment is designed to progressively build student competence</td><td>3</td><td>3</td><td>2</td><td>2</td><td>2</td></tr><tr><td>L9</td><td>Student work is subject to specified timetables and deadlines</td><td>3</td><td>3</td><td>3</td><td>3</td><td>2</td></tr><tr><td>L10</td><td>Courses are designed to support diverse learning styles and learner capabilities</td><td>3</td><td>2</td><td>2</td><td>1</td><td>2</td></tr></tbody></table>

Development category's weakness is the absence of systematization and formalization of processes on a wide scale. This includes such factors as lack of standards, low utilization of explicit plans for the guidance of course development, informal utilization and collection of information within management and optimization. Also inefficient distribution and utilization of produced e-learning resources is evident.

“Support provided in institutionally defined manner.”

“No incentives provided for innovative initiatives in most cases. The quality of planning and development is assessed formatively during some projects.”

“Information on effectiveness of support is utilized in development.”

“No standards in use for projects. Training provided in various forms for e-learning development. No systematic monitoring of compliance or value of procedures or principles.”
"Information on personnel e-learning skills not systematically and formally collected, therefore no explicit linkage between development of principles or standards."

"Utilization of existing e-learning resources is occasional. In addition, these resources or their usage are not systematically evaluated while planning the creation of new resources. No repository in use for the description and wide distribution of resources. Developing are currently."

"Ensuring of accessibility is not an institutionalized procedure. Accessibility is taken into consideration in some projects and initiatives. Accessibility has been recognized as focus of development, and actions have been taken."

<table><caption>**Table 4:** Maturity matrix of process category Development</caption><thead><tr><th colspan="6">Development: Processes surrounding the creation and maintenance of e-learning resources</th></tr></thead><tbody><tr><td>D1</td><td>Teaching staff are provided with design and development support when engaging in e-learning</td><td>4</td><td>2</td><td>3</td><td>2</td><td>3</td></tr><tr><td>D2</td><td>Course development, design and delivery are guided by e-learning procedures and standards</td><td>3</td><td>3</td><td>2</td><td>2</td><td>2</td></tr><tr><td>D3</td><td>An explicit plan links e-learning technology, pedagogy and content used in courses</td><td>2</td><td>3</td><td>2</td><td>3</td><td>2</td></tr><tr><td>D4</td><td>Courses are designed to support disabled students</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td></tr><tr><td>D5</td><td>All elements of the physical e-learning infrastructure are reliable, robust and sufficient</td><td>3</td><td>2</td><td>1</td><td>2</td><td>3</td></tr><tr><td>D6</td><td>All elements of the physical e-learning infrastructure are integrated using defined standards</td><td>3</td><td>2</td><td>2</td><td>2</td><td>2</td></tr><tr><td>D7</td><td>E-learning resources are designed and managed to maximize reuse</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td></tr></tbody></table>

Support category's reasonably high maturity is based on relatively versatile support functions available to use by personnel and students, and institutionalised approaches to continuous development of such functions. Although, teaching staff's personal e-learning pedagogical skills development could be more explicit part of strategic personnel capability development planning (HRD). In addition, more systematic approach to management and utilization of support specific information could lead to higher maturity. Library services provide extensive support and information related to essential information databases. ICT-support is built on standardized ITIL (Information Technology Infrastructure Library) practices, thus systematic approach to support ticket management is evident, including monitoring of support response times. E-learning technology support and training is continuously provided to teaching staff.
"Support is achievable through a variety of communication channels."

"Key support services utilize feedback largely within the development. Feedback collected from both, the students and the personnel, regarding the effectiveness and service of quality."

"Estimations for needed support resources are required and included in all the investment plans."

"Students' guide describes all the key support services for students."

"No formal or systematic collection of feedback from teachers regarding the effectiveness and clarity of student support services."

"The handling of student complaints is based institutionalized process and defined policies. Although, information related to the complaints is formally or systematically utilized in the development of e-learning."

<table><caption>**Table 5:** Maturity matrix of process category Support</caption><thead><tr><th colspan="6">Support: Processes surrounding the support and operational management of e-learning</th></tr></thead><tbody><tr><td>S1</td><td>Students are provided with technical assistance when engaging in e-learning</td><td>4</td><td>3</td><td>3</td><td>2</td><td>2</td></tr><tr><td>S2</td><td>Students are provided with library facilities when engaging in e-learning</td><td>3</td><td>3</td><td>4</td><td>4</td><td>2</td></tr><tr><td>S3</td><td>Student enquiries, questions and complaints are collected and managed formally</td><td>4</td><td>4</td><td>4</td><td>2</td><td>2</td></tr><tr><td>S4</td><td>Students are provided with personal and learning support services when engaging in e-learning</td><td>3</td><td>3</td><td>3</td><td>2</td><td>3</td></tr><tr><td>S5</td><td>Teaching staff are provided with e-learning pedagogical support and professional development</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td></tr><tr><td>S6</td><td>Teaching staff are provided with technical support in using digital information created by students</td><td>3</td><td>4</td><td>3</td><td>2</td><td>2</td></tr></tbody></table>

Evaluation category's low maturity is caused partially by the absence of formal and systematic approaches to internal assessment and audit of e-learning courses. In addition, there's no structured approach to measuring and controlling e-learning course quality, such as quality standards, guidelines and e-learning specific evaluation criteria.

"No information accessible on how personnel feedback is taken into consideration in planning and development of e-learning."

"Personnel feedback not collected systematically regarding their experience with e-learning technologies. Feedback is mainly occasionally and informally delivered."

"Lack of systematic distribution of knowledge regarding projects, in which e-learning solutions and resources are being developed."
"Approaches to quality management of projects and initiatives vary extensively, from informal to systematic and formal."

"No internal reviews or audits of current e-learning courses conducted."

<table><caption>**Table 6.** Maturity matrix of process category Evaluation</caption><thead><tr><th colspan="6">Evaluation: Processes surrounding the evaluation and quality control of e-learning through its entire lifecycle</th></tr></thead><tbody><tr><td>E1</td><td>Students are able to provide regular feedback on the quality and effectiveness of their e-learning experience</td><td>2</td><td>2</td><td>1</td><td>2</td><td>2</td></tr><tr><td>E2</td><td>Teaching staff are able to provide regular feedback on quality and effectiveness of their e-learning experience</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td></tr><tr><td>E3</td><td>Regular reviews of the e-learning aspects of courses are conducted</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td></tr></tbody></table>

Organization category's strength is based on institutionalized and strategic, systematic and formal, annual process of planning of development and allocation of resources. The e-learning specific issues could be notably more explicit in the process and in the plans that are defined and undertaken. E-learning development principles have been defined and documented, but not utilized throughout the institution. E-learning specific matters are not explicit in other key institutional principles or guidelines. The lack of systematic assessment of guidelines or principles is also evident.

"No systematic collection of staff feedback on guidelines or principles. Information received from previous development is normally utilized when developing e-learning strategy and guidelines of e-learning."

"Teachers participate in defining plans guiding development of e-learning, students much less."

"Not open and systematic sharing of information on or monitoring of failed development plans."

"No systematic feedback collection of technology specific aspects of courses. Information of students' e-learning capability is not collected or shared formally and utilization of this information within development is therefore challenging."
<table><caption>**Table 7:** Maturity matrix of process category Organisation</caption><thead><tr><th colspan="6">Organisation: Processes associated with institutional planning and management</th></tr></thead><tbody><tr><td>O1</td><td>Formal criteria guide the allocation of resources for e-learning design, development and delivery</td><td>4</td><td>3</td><td>2</td><td>3</td><td>2</td></tr><tr><td>O2</td><td>Institutional learning and teaching policy and strategy explicitly address e-learning</td><td>3</td><td>3</td><td>1</td><td>1</td><td>3</td></tr><tr><td>O3</td><td>E-learning technology decisions are guided by an explicit plan</td><td>3</td><td>3</td><td>2</td><td>2</td><td>2</td></tr><tr><td>O4</td><td>Digital information use is guided by an institutional information integrity plan</td><td>2</td><td>3</td><td>2</td><td>1</td><td>2</td></tr><tr><td>O5</td><td>E-learning initiatives are guided by explicit development plans</td><td>4</td><td>3</td><td>3</td><td>1</td><td>1</td></tr><tr><td>O6</td><td>Students are provided with information on e-learning technologies prior to starting courses</td><td>2</td><td>3</td><td>1</td><td>1</td><td>2</td></tr><tr><td>O7</td><td>Students are provided with information on e-learning pedagogies prior to starting courses</td><td>3</td><td>3</td><td>1</td><td>2</td><td>2</td></tr><tr><td>O8</td><td>Students are provided with administration information prior to starting courses</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td></tr><tr><td>O9</td><td>E-learning initiatives are guided by institutional strategies and operational plans</td><td>3</td><td>3</td><td>3</td><td>2</td><td>3</td></tr></tbody></table>

The characteristic of non-systematic and informal collection and utilization of information applies to all process categories, in the majority of the processes. Thus, the utilization of e-learning specific information in decision making and planning and execution of continuous development is challenging.

## 6 Conclusion

Evaluation of current state of processes is the basis for continuous improvement and development. Different kind of approaches on process assessment can be utilized to form an understanding of the gaps between current state and desired state. These approaches may cover both, quantitative and qualitative perspectives on the assessment of process performance.

Desired state of process performance is defined by the understanding of customer expectations and needs. In higher education institution's case, students and personnel, both, can be regarded as customers when designing and implementing improved processes. Documentation and systematic monitoring of process compliance are mechanisms for ensuring that the process design and actual execution do not differ dramatically.

Systematic and effective utilization of information in management is largely dependent on the availability and quality of the information. Management and development with insufficient information may lead to critically misleading decisions.
For example, allocation of resources between different development initiatives and programs must be based on reliable information and analysis. In order to build capability to more effective utilization of information and more effective creation and utilization of resources, an institution must create structures which support this. For example, processes, practices, tools, skills and policies for collection, sharing and utilization of meaningful information and resources should be promoted. Consistent and careful planning of development projects, and even more so, dissemination of results and best practices help an institution to function more effectively. Also experiences of unsuccessful projects can be of other projects' interests, as they help to address possible obstacles and risks which current projects can run into. In general, open sharing of all meaningful information and collaboration is one key success factor in organisational learning, and thus, organisational development.

The personnel's capability is one key aspect in successful development of e-learning. Therefore, enhancing personnel skills should cover e-learning specific aspects as well. To support this, the assessment of e-learning skills should be included in human resource development. Students – whether regarded as customers or not – should also be given a chance to take part in development. Their views are valuable, and help to understand the desired performance and quality of execution. In addition, students have skills that may prove to be beneficial for the development. The strategic management team plays a significant role in the acknowledgement of the importance and opportunities of e-learning: how firm the basis for the development is, depends heavily on the mandate the strategic management team is willing to provide the institution and its developers with.

Assessing e-learning maturity with eMM is an intensive and resource demanding process that benefits from the participation of people from different domains. As with this case study, it was considered beneficial and crucial that representatives from strategic management, quality assurance, e-learning pedagogy, e-learning support and e-learning technology development were involved in the process. Also, student involvement in the process is something to consider carefully in future re-assessments of maturity.

Even though institutional culture would rather promote different kind of approaches to assessing and developing processes, introducing oneself with eMM can widen perspectives on the field. In the case institution, eMM CORE proved its value. Currently, several different scale development initiatives and projects have been planned and started, which can be linked to the results of the study. The purpose is to re-assess process maturity – at least on process categories and processes which were chosen as objects of current development – in 2015. Currently, the strategic management team has acknowledged the ever increasing role of e-learning. This has led to allocation of larger resources for the development of e-learning throughout the institution. Re-organization and re-defining the mission of e-learning department is expected to lead to higher quality processes and higher capability. Initial actions have been taken to develop more effective evaluation practices of e-learning, more specifically e-learning environments, processes and outcomes. Definition of institution-wide e-learning course quality criteria and development of practices and policies for internal evaluation of e-learning courses should lead to higher maturity and capability on evaluation -category.
Future steps of the research will broaden the scope and continue to focus on the quality management and process oriented approaches to organizational development. Business process management as a framework for higher education institution's continuous process improvement is an area that holds interest for further research as well. Performance improvement activities are becoming increasingly more crucial in public higher education, in Finland as well, due to increasing competition and reformed financing policies. Thus, structured and systematic approaches to development are needed more than ever.

## References

1. Ruohonen & Multisilta. Preface, xiv. In *Nicholson, P., Thompson, J.B., Ruohonen, M. & Multisilta, J. (eds.). E-Training Practices for Professional Organisations*. Kluwer Academic Publishers, Boston, MA (2005).

2. Marshal, S. What are the key factors that lead to effective adoption and support of e-learning by institutions? HERDSA, Sydney, Australia (2008), from http://www.voced.edu.au/node/25701

3. Haukijärvi, I. & Nevalainen, T. Developing a Quality Enhancement Framework for Collaborative Online Courses - Building on Constructivism with a Design Science in Information Systems Approach. In *Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2014* (pp. 480-489). AACE, Chesapeake, VA (2014).

4. Helfat, C. E., & Finkelstein, S., & Mitchell, W., & Peteraf, M. A., & Singh, H., & Teece, D. J., & Winter, S. G., Dynamic Capabilities. Understanding Strategic Change In Organizations. Blackwell Publishing, Massachusetts, MA (2007).

5. Hammer, M. What is Business Process Management? In *J. Vom Brocke & M. Rosemann (eds.), Handbook on business process management: introduction, methods and information systems 1st Edition* (pp 3-16). Springer Verlag, New York, NY (2010).

6. Mutafelija, B., & Stromberg, H. Systematic Process Improvement using ISO 9001:2000 and the CMMI. Artech House, Massachusetts, MA (2003).

7. Kasse, T. Practical insight into CMMI. Artech House, Boston, MA (2008).

8. Marshal, S. eMM 2.3 Process Descriptions. Victoria University of Wellington, Wellington, New Zealand (2007), from http://www.utdc.vuw.ac.nz/research/emm/Publications.shtml.

9. Hevner, A. R., & March, S. T., & Park, J., & Ram, S. Design science in information systems research. *MIS Quarterly*, 28 (1), 75-105 (2004).