# Finding a School Technological Partner

Teresa Pereira, António Castro

► To cite this version:

Teresa Pereira, António Castro. Finding a School Technological Partner. 10th Next Generation of Information Technology in Educational Management (ITEM), Aug 2012, Bremen, Germany. pp.131-141, 10.1007/978-3-642-38411-0_12. hal-01468469

HAL Id: hal-01468469

https://inria.hal.science/hal-01468469

Submitted on 15 Feb 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Finding a School Technological Partner:

A Multi-criteria Method for School Information System Producer
Selection

Teresa Pereira¹ and António Castro²

¹Escola Superior de Estudos Industriais e de Gestão, Instituto Politécnico do Porto, Portugal
teresapereira@eu.ipp.pt

²Algoritmi Center, Universidade do Minho, Portugal
Antonio.castro@algoritmi.uminho.pt

**Abstract.** The choice of an information system is a critical factor of success in an organization's performance, since, by involving multiple decision-makers, with often conflicting objectives, and having several alternatives with aggressive marketing makes for particularly complex decision-making in terms of choice by consensus. The main objective of this work is to create an analysis and selection method for an information system designed to support school management, pedagogical and administrative components, using a multi-criteria decision aid system – MMASSITI – Multi-criteria Methodology to Support the Selection of Information Systems/Information Technologies. It integrates a multi-criteria model that seeks to provide a systematic approach to the process of choice of information systems, able to meet sustained recommendations concerning decision and scope. Its application to a case study has identified relevant factors in the selection process of school educational and management information system and how to gain a solution that allows the decision maker to compare the quality of the various alternatives.

**Keywords.** Decision support system; multi-criteria decision analysis; information management systems; educational management.

# 1 Introduction

A school information system is seen as a vital resource for organizational support, and its proper management is essential, both at the strategic, tactical and operational levels. The existence of a tool to support decision-making in relation to the choice of information systems/information technologies (IS/IT) in an organizational context is thus of great importance.

In this paper, the problem is to address the needs for choice and selection of a school management information system (SIGE), to be able to follow the architecture designed for monitoring schoolwork. It is also intended that this SIGE demonstrates an evolutionary capacity, to follow the implementation needs inherent to internal and external environmental changes.
The architecture of SIGE is already set and the purpose of this work is the selec-
tion of both a SIGE and a partner that is already acting in the educational management
environment and best suits the architectural model proposed. The selected partner
should also be able to leverage and develop its information system architecture in a
convergent form with the model.

The process of selecting is complex and involves several dimensions: i) knowing and bearing in mind the needs framed in the architectural model - informational architecture of the school in functional terms, the strategy of its information system, organization and global strategy and operational planning of the school; ii) implementing a process that allows for systematic evaluation and is consistent, with extended alternatives that have a high number of features and attributes relevant to the evaluation process; and iii) managing a set, more or less broad, of conflicting objectives. Additionally, this process involves, systematically, various decision makers (a team of research and IS/IT development and computer operators in the school area) with different point of views and divergent interests.

The main objective of this work is to develop a comparative study of four SIGEs that are operating on the Portuguese market and which involves: i) a clear structuring of the problem, namely: a coherent and consistent family of criteria, identification of features and functionalities that should be incorporated in the decision-making process; and ii) the definition of an approach that enables objectively the comparison of the quality of different alternatives. For supporting the decision aid process, we used a multi-criteria decision support system (DSS) developed by Pereira [1], MMASSITI – Multi-criteria Methodology to Support the Selection of IS/IT, which seeks to provide a systematic approach to the multi-criteria decision process, capable of producing sustained recommendations with regard to the solution adopted in view of the particular context of education and the environmental context.

This paper is organized as follows: in section 2 an architectural model framework defines the functionality that an integrated school management system (SIGE) should cover, or possess; in section 3, in summary, the multi-criteria model incorporated into the SAD-MMASSITI is shown; in section 4 a consistent family of criteria for the specific Portuguese school context of decision aid in accordance with the methodology used is defined; in section 5 the results of the application of this methodology to the study of a case are presented and discussed; and in section 6, we present the main conclusions drawn from the application of the SAD - MMASSITI to the selection of a SIGE to support the research and development of a school information management system project.

## 2 Architectural model for a school management system

Schools in Portugal are units of an educational system with great complexity. They are different from each other, independent and with great autonomy. However, the technological evolution journey supports educational management in this development through progressive transformation. In our educational management research, an adaptation of a model for informational architecture of school management was creat-
ed – The Viable System Model (VSM) of Stafford Beer [2]. The goal is to develop school management to provide educational management.

Before going to the selection of a technological partner to this research program, it is useful to describe the architectural model docket. The architectural model adapted from the VSM has five elements, is recursive and adaptable to change. It consists of a recursive structure to obtain the information produced in different operations – in what is taught, in what students learn and in the environments related to the school. This model has five elements: (i) operations; and their (ii) coordination; (iii) monitoring and control; which is able to produce (iv) intelligence; supporting school environments adaptable to established (v) policies.

**Sub-system 5: Politics.** The executive team and the school director have the responsibility to set politics and strategic orientation for the school. This board produces strategic documents describing the school educative project, school curricular project and school regulation book. The political definition is supported by the information system; from its operations data are collected and used to produce management maps. Without an automated procedure, school decision-making is strongly influenced by the intelligence element in sub-system 4.

**Sub-system 4: Intelligence.** This sub-system is responsible for the production of management maps. It is based on school-work assessment indicators in pedagogical and administrative tasks. It is also responsible for the environment's interaction. In this element, information required by the Ministry is produced and delivered.

**Sub-system 3: Control and monitoring.** In the Portuguese educational system, audit and control are provided by the General Inspectorate of Education (IGE). But elements from this sub-system can be used to keep a check on the internal environment. Examples include attendance control systems, and the school dropout control system.

**Sub-system 2: Coordination.** In the coordination support sub-system some automatized processes for the coordination of study cycles are provided, to departments and teacher groups. In this element, coordination tools are also included, such as teacher timetables, class timetables, laboratory use timetables and classroom timetables.

**System 1: Operations.** Teaching activities and all teaching system support items, including plans, daily records and assessment items are included here. All daily schoolwork is recorded as data in this element.

**Communications.** The way each VSM element works depends a lot on the communications process, between elements, and between elements and environments. There are four communication domains in this VSM adaptation that trade information between the management element, the operation and environment elements. The tools to do this must have capacities to broadcast and to migrate information as well. School communication channels are distinct and have different functions. They exist with
broadcast and directional functions and are vehicles to communicate between VSM
elements, making connections in the internal as well as the external environment.
Upward communication presumes a form of reduction, but constitutes a vital element,
structuring and guaranteeing autonomy and viability. Schools need to use institutional
e-mail, newsletters, have presence on the web, and use social networks and Moodle
(in specific places). The technological partner must have the capacity to aggregate
them.

3 MMASSITI DSS

Though a large number of multi-criteria decision analysis (MCDA) methods have been proposed by researchers, none can be considered as ‘The Method’ appropriate to all decision making situations. All of them require a facilitator/analyst to perform them. Most of the problem solving in IS/IT selection uses the most popular and quoted MCDA methods, such as ELECTRE outranking method family [3,4], PROMETHEE outranking method family [3,4], and the AHP method [5] among others, to access the aggregation phase.

The decision aid support process generally involves incomplete, inaccurate information, multiple criteria and is a group decision problem or involves different decision-making agents.

The existing multi-criteria methodologies are based on the elicitation of values of consequences on twig-level criteria or attributes and also on the relative weight of criteria. Belton and Ackermann [6] emphasize the fact that the work done in the area of MCDA has been predominantly based on methods to support evaluation process and choice, instead of structuring the problem.

Specific procedures, based on cognitive mapping, such as COPE and V.I.S.A [6], SSA - Soft Systems Analysis [7 - 9], or Keeney's value focused thinking [10] have been used to address the structuring phase coupled with the most popular MCDA methods worldwide. The structuring phase, and consequently the definition of a coherent and consistent family [10 - 12] of criteria is crucial for the success of any multi-criteria methodology because criteria will be the key elements upon which global preferences will be set up [13].

We propose a multi-criteria decision aid (MCDA) model, translated into a Decision
Support System (DSS) named "Multi-criteria Methodology to Assess and Select In-
formation Systems/Information Technologies (MMASSITI)". This proposed method-
ology differs from others in that it already includes the problem structuring phase as a
predefined consistent family of criteria (essential, independent, controllable, opera-
tional, decomposable, non-redundant, complete, measurable, concise, understandable)
applied to an IS/IT selection to be validated and to adjust for the given context to be
applied. In this methodology, each criterion has a description and a measure suggest-
ed, so that any decision maker (DM) understands the criterion in the same way, fos-
tering the DM's judgment. This initial set of criteria is basically a starting point for
discussion; it aims at involving all the DMs, giving them a common understanding of
each criterion and helping them to start using the methodology. It also helps the DM
to define the family of criteria that will be used in context (from the predefined set of
criteria), validating the properties required for this family to be consistent and coher-
ent

MMASSITI is a DSS that embodies a methodology proposed by Pereira [1], to support the IS/IT selection in an organizational context. It has been designed to be easy to understand and use, without a specific need for a decision making expert, to offer the DM an effective support decision-making tool and to act as enhancer of the specification accuracy. With this purpose the methodology is simple and guides the group decision makers (GDM) throughout. It is presented "as a check-list" - an exhaustive and coherent family of criteria (shown in Figure 1).

<figure><img src="image_2.png" /><figcaption>**Figure 1.** Multi-criteria Model as an exhaustive family of criteria.</figcaption></figure>

This family of criteria does not address a specific IS/IT, but, rather generally co-
vers all the criteria, taking into account the choice of any IS/IT in an organizational
context. The intention is to present to the GDM a "starting point" and guides them
towards thinking about the importance (or the lack) of a particular criterion to the
specific problem and selection scope, whether it is explicitly and fully defined, so that
a common understanding exists. It is the GDM that, in the end, defines and validates a
consistent and coherent family of criteria for the context to which it is applied, by
restricting, modifying, or adding new criteria to the initial family of criteria they are
presented to begin with. This consistent and coherent family of criteria defined by
GDM should represent all the different and relevant aspects of the problem and
should present these properties: independence; measurability; operationality; under-
standability; and non-redundancy.

As an IS/IT is characterized by its functionalities and entities, which sometimes
overlap, the multi-criteria model is divided into two phases: phase 1, concerns macro
aspects of the IS and its suitability to the organization strategies, business, resources,
requirements and alignment with existing IS/IT; phase 2, concerns micro aspects of the IS/IT itself, such as the technical and functional specifications and requirements. Concerning the decision making problem and the alternatives for analysis, the methodology could be carried out by joining both phases 1 and 2, or with phase 2 only, if the number of alternatives is limited and, at the same time, if all of them match the macro level.

Figure 2 illustrates the model used in MMASSITI, which takes into account all organizational aspects (phase 1 – macro level) and the IS/IT technical issues (phase 2 – micro level).

<figure><img src="image_3.png" /><figcaption>**Figure 2.** Multi-criteria Model.</figcaption></figure>

Considering these issues, individual preferences must be reduced to a collective preference for group decision making to take place [14 - 16]. Thus, the preference of the GDM is assumed as a collective preference or consensus throughout the methodology.

The MMASSITI DSS performs several steps in sequence, to guide the GDM through the multi-criteria methodology.

* First step: define the consistent family of criteria in consensus with the GDM (to do that a full description and specification of the decision scope must be done);

* Second step: analyze/add and validate each criterion description by the GDM until a consensus is reached;

* Third step: set up the definition of a “neutral” IS/IT and of a “better” IS/IT in the business and organizational context of the analysis;
* Fourth step: set up the collective relative importance ranking (weight) as- signed by the GDM to each criterion according to the swing weight proce- dure [17];

* Fifth step: Define a continuous scale with seven semantic correspondence levels ("Much Worse", "Worse", "Slightly Worse", "Neutral", "Slightly Bet- ter", "Better" and "Much Better"). Two of them are reference levels to eval- uate each alternative on each criterion: the "Neutral" level and the "Better" level. The "Neutral" and "Better" level definitions by the GDM are mandato- ry. This interval scale is fully defined by the GDM. It is a fixed scale that will be applied to all alternative evaluations on each criterion, in both phases.

* Sixth step: Adjust the “Neutral” IS and “Better” IS definition for each crite- ria.

* Seventh step: Assess each alternative for each criterion. To do so, first the GDM must be aware of the existing information about each alternative per criterion, even if it is imperfect information (such as, ask the provid- ers/supplier of each alternative to do IS/IT presentations and give additional information, to analyze the provider/supplier portfolio, to see, if possible, each alternative in a real context, etc.). Consensually, for each criterion, the GDM must attribute a semantic level to each alternative, taking into consid- eration the two reference levels, and assign a collective value in accordance with the previously-defined continuous semantic scale.

* Eighth step: Use an additive model to rank each alternative.

* Finally, MMASSITI presents the IS/IT ranking order and respective score value.

MMASSITI also provides to the GDM, a sensitivity and robustness analysis. At the end, the GDM provides effective support for choosing the “best” IS/IT concerning the issues to be addressed and according to existing information at the time.

# 4 Case Study: SIGE assessment analyses and selection

The case study shows an implementation of a decision aid selection process using the DSS MMASSITI to choose a SIGE from four existing SIGE alternatives on the mar- ket, coded A, B, C and D. The information required about each SIGE, to perform the decision aid selection process, was obtained in the region’s schools, which use the different SIGEs, and additionally the available information (features and functionaliti- ties) on the webpages of suppliers. The multi-criteria analysis was performed only with phase 2 of the methodology, considering specific aspects of the SIGE, since any one of four SIGEs were intended for use in the same context – educational manage- ment. This analysis was made by three decision makers.

## 4.1 Structuring the Problem – family of criteria

The definitions of a coherent and consistent family of criteria were produced and some of the predefined criteria of DSS MMASSITI were adjusted and added to the
relevant criteria that we thought were crucial to the selection process of a SIGE. Criteria that made no difference to the four SIGE evaluations were deemed irrelevant and therefore not considered.

Seven criteria were selected: cost or licensing of the SIGE; MISI certification (Portuguese ministry of education certification); required functionalities; modularity; web environment; user friendliness; and intuitive navigation (see Figure 3).

<figure><img src="image_4.png" /><figcaption>Figure 3. Criteria definition.</figcaption></figure>

For the assessment of each criterion of each SIGE it was necessary to create its description (see Table 1), so that decision makers had a common basis of understanding. The criteria were sorted in descending order of importance and weights were assigned using a swing weights procedure [18], both by consensus of the decision makers.
<table><caption>**Table 1.** Operationalization and assignment of weights to the family of criteria.</caption><thead><tr><td>Criteria</td><td>Description</td><td>Weight</td></tr></thead><tbody><tr><td>MISI Certification</td><td>The entire educational management software has to send information to the Ministry. This process has its own formats. Only the Office for the Coordination of Information Systems of the Ministry is able to assign this certification process</td><td>100</td></tr><tr><td>Cost of acquisition or licensing</td><td>Costs of the solution. Considered as annual licensing and maintenance</td><td>90</td></tr><tr><td>Web Environment</td><td>Solution's ability to operate across the web, whether on an intranet (internal network) or on the internet</td><td>80</td></tr><tr><td>Funcionalities</td><td>Four functional areas with technological support are considered: students; student welfare and social services; accounting; and staff</td><td>60</td></tr><tr><td>Modularity</td><td>Ability to time phase in the full implementation of an integrated solution, in line with the priorities and existing human and financial resources, as each module has well-defined functions</td><td>55</td></tr><tr><td>User friendliness</td><td>Design with aesthetic sense, capable of representing the necessary actions and generate interest by the users</td><td>45</td></tr><tr><td>Intuitive Navigation</td><td>Ease of use of SIGE in obtaining and managing content</td><td>40</td></tr></tbody></table>

## 4.2 Articulation and modeling of preferences

After the selection of the consistent family of criteria and their ranking, it is necessary to define the levels of attractiveness (see Figure 4), with two of them being mandatory: the “Neutral” and the “Best” levels.

“Neutral” and “Best” levels, respectively, have been defined as: “the SIGE with the basic functionality in school administration and pedagogical areas, with at least a partial MISE certification (one module). Developed in a language that allows them to be used on the web (e.g. using php or asp), with a pleasant aesthetic interface that provides intuitive navigation facilities. Allowing a modular acquisition and with module integration capacity within the same supplier”; and “SIGE with full MISE certification that provides functions across administrative and pedagogical areas, which provide communication services (inside and outside of school), functions in support of the teaching process and learning support service 24 hours for 7 days a week, with navigation facilities and total support, with the possibility of modular acquisition and allowing third-party module integration”.
<figure><img src="image_5.png" /><figcaption>Figure 4. Valuation levels of attractiveness.</figcaption></figure>

## 4.3 Aggregation of the valuation of alternatives for each criterion

After the valuation of each criterion of each SIGE, the additive aggregation led to the results shown in Figure 5. The SIGE C was the best ranked with 85.73 points, followed by SIGE B with 24.82, SIGE D with 23.76 and SIGE A with 11.74 points.

<figure><img src="image_6.png" /><figcaption>Figure 5. Aggregation results of each alternative for each criterion.</figcaption></figure>
The sensitivity and robustness analysis was also performed to verify the consisten-
cy of the results. It turns out that the assignment of weights was equal for all the crite-
ria, not changing the ranking of each SIGE.

5 Conclusions

A multi-criteria analysis translated into the SAD- MMASSITI was used in the com-
parative analysis of four SIGEs, taking into account: an architectural model frame-
work; the functionalities defined as the ones that an integrated school management
system (SIGE) should cover; the information available on the webpages of the SIGE
suppliers; and the experience of three decision makers who performed the analysis.

The case study was carried out taking into account the intended architectural mod-
el for a real school context but was not specific, needing to cover the present and fu-
ture reality of administrative and pedagogical school management needs. The valua-
tion, assigned to the alternatives in the analysis, resulted in the classifications of sys-
tems, understood as a result of the experience and a quantitative and qualitative analy-
sis of DMs using information available. The aim was to demonstrate that the process
of analysis and selection of a SIGE supported by a MCDA, despite the time consumed
in the analysis and application of the model, allows the systematization of characteris-
tics and functionalities considered relevant in this kind of system also integrates the
priorities of decision makers involved in the evaluation and selection process, making
the decision-making process clearer and more objective. In addition, it was found that
the SAD used allows the decision maker to systematize the process of analysis, which
is particularly relevant when the decision problem is complex. The special character-
istics of the SIGEs, considered at the start of the analysis, forced additional criterion
to be incorporated in the SAD-MMASSITI (facilitated by the flexibility of the model)
in order to provide a more careful study of educational management support.

References

1. Pereira, T.: Metodologia Multicritério para Avaliação e Selecção de Sistemas de Informação para a Industria. PhD Theses, University of Minho, Portugal (2003)

2. Beer, S.: The Viable System Methodology and Pathology. Journal of the Operational Research Society, 35(JSTOR), 7-25, (1984)

3. Roy, B.: Méthodologie Multicritère D'Aide à la Décision. Economica (1985)

4. Roy, B., Bouyssou, D.: Aide Multicritère à la Décision: Méthodes et Cas. Paris (1993)

5. Saaty, T.L.: The Analytic Hierarchy Process. McGraw-Hill, New York, NY (1980)

6. Belton, V., Ackermann, F.: Integrated Support from Problem Structuring through to Alternative Evaluation Using COPE and V.I.S.A.. Journal of Multi-Criteria Decision Analysis, 6, 115-130, (1997)

7. Checkland, P.: Systems Thinking, Systems Practice. JohnWiley & Sons (1981)

8. Checkland, P., Scholes, J.: Soft Systems Methodology in Action. John Wiley & Sons (1990)
9. Checkland, P., Holwell, S.: Information management and organizational processes: an approach through soft systems methodology, Journal of Information Systems, 3 (1), 3-16, (1993)

10. Keeney, R.L.: Value-Focused Thinking: A Path to Creative Decision-Making, Harvard University Press (1992)

11. Keeney, R.L., Raifa, H.: Decisions With Multiple Objectives: Preferences and Value Tradeoffs, John Wiley & Sons (1976)

12. Roy, B., Bousyssou, D.: Procédures d'agrégation multicritère conduisant à un critère unique de synthèse. LAMSADE - Laboratoire d'Analyse et Modélisation de Systemes pour l'Aide à la Decision, Université de Paris-Dauphine, France (1987)

13. Vincke, P.: Multicriteria Decision-aid. John Wiley & Sons, New York, NY (1992)

14. Goodwin, P., Wright, G.: Decision Analysis for Management Judgment. John Wiley & Sons (1991)

15. Kim, S.H., Ahn, B.S.: Group decision making procedure considering preference strength under incomplete information. Computers and Operations Research, 24, 12, 1101-1112, (1997)

16. Kim, S.H., Choi, S., Kim, J.: An interactive procedure for multiple attribute group decision making with incomplete information: Range-based approach. European Journal of Operational Research, 118, 139-152 (1999)

17. Winterfeldt, D.V., Edwards, W.: Decision Analysis and Behavioural Research. Cambridge University Press, New York, NY (1986)

18. Weber, M., Borcherding, K.: Behavioral influences on weight judgements in multiattribute decision making. European Journal of Operational Research, (67), 1-12, (1993)