# User Experience Evaluation in eModeration: The Case of Higher Education Institutions in Developing Countries

Corné Staden, Jan H. Kroeze, Judy Biljon

► To cite this version:

Corné Staden, Jan H. Kroeze, Judy Biljon. User Experience Evaluation in eModeration: The Case of Higher Education Institutions in Developing Countries. 18th Conference on e-Business, e-Services and e-Society (I3E), Sep 2019, Trondheim, Norway. pp.621-633, 10.1007/978-3-030-29374-1_50 .
hal-02510140

HAL Id: hal-02510140

https://inria.hal.science/hal-02510140

Submitted on 17 Mar 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# User Experience Evaluation in eModeration: the case of Higher Education Institutions in Developing Countries

Corné J. van Staden[0000-0002-3991-3128], Jan H. Kroeze[0000-0001-7118-4853] and Judy A. van Biljon[0000-0002-4646-1641]

School of Computing, University of South Africa, Science Campus, 28 Pioneer Ave, Florida
Park, Roodepoort, 1709, South Africa

vstadcj1@unisa.ac.za, kroezjh@unisa.ac.za, vbiljja@unisa.ac.za

**Abstract.** While the need for the moderation of examination scripts as part of quality assurance is globally undisputable, moderation associated risks and challenges are more pronounced in developing countries. eModeration (online moderation of examination scripts) can improve the user experience of an examination moderation process while lowering the risk of losing scripts and delaying the moderation process. Various factors could contribute to resistance against implementing and adopting eModerate systems in Higher Education Institutions, ranging from human factors to technical issues and organisational resistance to change. The focus of this study is on the human factors involved in eModeration, i.e. the factors influencing the user experience when using eModeration. The research uses a design science research methodology, which includes the design, development and testing of a User Experience Evaluation Framework for eModeration. The contribution of this paper is the demonstration of how eModeration (e-Service) is relevant to Higher Education management, as well as the provision of some insights regarding eModerators’ user experience of an existing system. The resultant artifact is a validated User Experience Evaluation Framework for eModeration, which can be used to improve human experience of electronic moderation with an emphasis on improving the quality of educational assessment practices in developing countries.

**Keywords:** e-Service Moderation, user experience, usability.

## 1 Introduction

The role of computing has changed over the years, especially in the Higher Education (HE) environment. There are indications of an emerging trend in research regarding electronic moderation, but Higher Education Institutions (HEIs) face certain challenges with the implementation and use of systems for online moderation of examination scripts (eModeration) ranging from technical obstacles regarding internet access, file size and infrastructure problems to human factors. More specifically, HEIs are faced with the challenge of finding eModeration systems, which provide an acceptable user
experience level within the constraints of a developing country such as South Africa where moderation is an integral part of ensuring standards.

The traditional method of moderation of examination scripts involves the distribu-
tion of hard copies of the actual marked examination scripts to moderators using courier
delivery or postal services, which is expensive, time consuming and not always effec-
tive [1]. The motivation for this study stems from HEIs' need to consider alternative
modes of distribution when they employ Information Communication Technology
(ICT) as an e-Service solution towards improving effectiveness, efficiency and resili-
ence. Resilience, namely the ability of systems to cope with external shocks and trends
[2] is an important consideration in HEI's. User experience is a critical part of making
any information system - including eModerate systems used in HEIs - more acceptable
and resilient, but the evaluation of these eModeration systems are lacking. Therefore,
the research question for this paper is: how will the implementation of a User experi-
ence Evaluation Framework for eModeration influence eModeration practices (e-Ser-
vices) within the context of Higher Education Institutions in South Africa? This paper
addresses the knowledge gap by reporting on the empirical evaluation of a User Expe-
rience Evaluation Framework for eModeration implemented in South Africa. The con-
tribution is an updated eModeration Framework together with new insights on the chal-
lenges of implementing a resilient eModeration system in a developing country, which
is moving away from traditional methods of moderation. The e-Service under discus-
sion in this paper will be that of an eModerate system while the e-Society will be man-
agement and eModerators in HEIs.

The paper is structured as follows: the literature review is followed by sections on
the methodology and the research results respectively. The findings are discussed and
the resultant framework is explained before presenting the contribution of this paper.

2 Literature Review

2.1 eModeration

In contrast to online assessment and automated marking, which have been researched
in depth and applied in various HEIs successfully, electronic examination script mod-
eration is still a relatively new phenomenon, and related software applications are lim-
ited and not widely used. This paper focuses on this relatively under-researched area.

Different online assessment technology tools exist which allow HEIs to assess their
students making use of electronic technology [3]. Although little effort has been made
to adopt these innovative technologies where HEIs are still using traditional assessment
methods [4], the need to enhance student learning in tertiary institutions is playing a
role to accelerate the conversion from paper-based to digital methods [4, 5]. Similarly,
the need to optimise moderation processes prompts the transformation from traditional
paper-based moderation methods to electronic examination script moderation in HEIs.
Digital systems could overcome the challenges that are typical of a developing country,
such as slow and unreliable postal services, expensive courier systems, the involvement
of distant international experts and feedback to students enquiring about their marked
scripts.
The concept of eModeration is used in various contexts, for example, mentoring and discussions between lecturers and students on digital platforms [3, 6-12]. Adie’s studies [13] on social moderation focus on lecturers acting as eModerators who purposefully develop agreements on standards, quality and consistency of assessment judgement across different programmes. Adie [14] further proposes a theoretical framework for online professional discussion. Grainger, Adie and Weir [15] use social moderation meetings to discuss the methods which members are supposed to use during assessments, for example, using a criteria sheet to ensure a common understanding of accountability and justification, as well as to build a community. Adie [16] recommends that moderation forums can also be used in online moderation meetings to support the collaborative professional development of teachers and the formation of a common understanding of quality in students’ work.

Moderation forums focus on sharing meaning where, for example, more than one marker is used to mark the same assessment to ensure consistency and to facilitate discussions between lecturers and markers [16, 17]. In this paper eModeration refers to a specific type of peer review, defined as: "the electronic moderation (quality assurance/critical reading) of summative examination scripts by external moderators in a virtual learning environment" [18]. Within the context of peer moderation, electronic moderation of examination scripts involves the use of some electronic moderation systems to electronically moderate examination scripts [19, 20]. Peer electronic moderation involves different users, for example, internal examiners (users who grade the assessment initially) and external examiners (users who grade the assessment externally – eModerator), as well as management who can deans or examination officers involved in the moderation process. The focus of peer moderation is not between lecturer and student but between external examiners and management. External examiners judge the internal examiners’ grading using an eModerate system. Management oversees the process of eModeration using the eModerate system to track the progress of the moderation process, to peruse the feedback received from the external moderators, and to intervene and take action where necessary. The e-Service under discussion in this paper will be that of an eModerate system, while the e-Society will be management and eModerators in HEI. The user experience that is under investigation is that of both the eModerator and management.

An eModerate system is supposed to provide the user with an interface that can be used to electronically assess and re-grade marked examination scripts of students at HEIs. The marked scripts that need to be moderated should be scanned and uploaded from the original, hand-written examination scripts of students with the internal markers’ allocated marks. The eModerate system should allow the institution requesting moderation to upload the scanned scripts to, for example, a module site in a virtual learning environment, where the eModerator will have secure access to the documentation. Once the eModerator has finished with the electronic control (confirmation or re-grading) of the marked scripts, he/she must be able to upload the documents back onto the same system for the Higher Education Institution (HEI) requesting the moderation to download and view. Some advantages of using an eModerate system are the ability to track the moderation process; cost effectiveness; the fact that it can be done
anytime and anywhere; and the electronic availability of moderated examination script(s) for future reference [20].

Not only is it important to complete the task of eModeration in a flexible, efficient and cost-effective way, but it is also important to evaluate the users’ experience of the eModerate system used to moderate examination scripts electronically. According to Preece, Sharp and Rogers [21], user experience is complex and draws from different fields, such as interactive design, information architecture and usability, amongst others. eModerate systems allow users to interact with the system at any time and place. According to the International Organization for Standardization’s standard 9241-210 (human-centred design for interactive systems), user experience is described as “[a] person’s perceptions and responses resulting from the use and/or anticipated use of a product, system or service. User experience includes all the user’s emotions, beliefs, preferences, perceptions, physical and psychological responses, behaviours and accomplishments that occur before, during and after use. User experience is a consequence of brand image, presentation, functionality, system performance, interactive behavior and assistive capabilities of the interactive system, the user’s internal and physical state resulting from prior experiences, attitudes, skills and personality, and the context of use” [22].

A particular challenge regarding user experience relates to how the measurement of all instrumental and non-instrumental aspects or qualities are associated with the design process [23, 24]. The measurement of instrumental and non-instrumental aspects will provide feedback on the user’s use and acceptance of products or services, referred to as the user’s emotional reaction to the product or service. Another aspect of user experience concerns the situation in which a product or service is used [8, 24-27]. Roto [28] extends the scope by including factors such as infrastructure, services, people and the technology context. The process used to improve the usability of an artifact involves an iterative design cycle [24] which promotes effectiveness and efficiency of the task to be performed, as well as the satisfaction of the user [29]. Usability goals are objective [21, 30], while user experience aspects are more subjective [21, 22].

There has been a shift in determining a product’s success from only considering usability aspects to including aspects such as product interaction, individual disposition and context, which in turn affect the user experience of a particular product [31]. For the purpose of this paper, usability will be viewed as a concept embedded in user experience [32]. The researchers also view satisfaction as a shared attribute of both user experience and usability. User experience involves more than a product’s utility and usability — the subjective nature of user experience is affected by the user’s internal state and, furthermore, the context and the perceived image of the product’s instrumental and non-instrumental qualities [23, 33-36]. According to Tullis and Albert [37] user experience includes three observable and measurable characteristics: the “user” who is involved, the user who is “interacting with a product or system” via an interface, and the user’s “experience”.

Ensuring effective and efficient moderation practices in HEI’s is a challenge in developing countries where resource constraints impact the effectiveness, efficiency and safety of examination scripts’ distribution and storage systems. According to Abdelnour-Nocera and Densmore [38] information and communication technologies for
development (ICT4D) as a field is concerned with the use of ICT to support poor and
marginalized people in social, political and economic spheres. Many HEI's in develop-
ing countries operate under financial and knowledge resource constraints and thus fall
in the application domain for ICT4D research.

ICT can be used as a platform for development [39] and resilience is the key to the
sustainability of ICT systems in developing countries [40]. Resilience has three primary
foundations namely robustness, self-organisation and learning and six secondary ena-
blers namely redundancy, rapidity, scale, diversity, flexibility and equality [2, 41]. This
study supports resilience through the conversion of a manual system where scripts
could get lost or stolen in transit to eModeration which greatly reduces the risk of losing
scripts. The resulting framework was implemented and evaluated in a developing coun-
try and can thus be used to assist managers and users to evaluate the user experience in
order to ensure sustained moderation and consistently high tertiary standards.

The constructs associated with user experience is measured by non-instrumental (non-task-orientated usability) qualities and instrumental (task-orientated user experience) qualities [20]. The proposed User Experience Evaluation framework also considers the eModerators' willingness to adopt to eModeration and produce digital content.

The constructs associated with user experience [23, 28, 42-43] were mapped onto eModeration aspects, which guided the design and development of the questionnaire used during the survey. The data gathered during the survey was used to “determine which user experience constructs would be relevant” with regard to a User Experience Evaluation Framework for eModeration [20]. Through the literature review and research process the following key constructs were taken into consideration in terms of eModeration user experience for the study: system (eModeration web application), context (eModeration in Private Higher Education Institutions (PHEIs)) and user (eModerator and management) [19, 44]. The following theoretical approaches of other relevant studies were applied in this study: the concepts and principles used in user experience frameworks in the fields of user experience of audio players [33], interacting with products in an online environment [45] and mHealth [46]. The principals used by these indirectly related frameworks guided the design and development of the conceptual User Experience Evaluation Framework for eModeration [1]. After the design of the concept framework and presentation to the academic community the framework was used in the context of PHEIs to ensure that the framework will support academic assessment practices related to moderation.

## 3 Research Approach

The Design Science Research approach used in the study is an adaptation of Hevner's conceptual framework for Design Science Research in Information Systems [47]. This approach was appropriate for the design and development of a User Experience Evaluation Framework for eModeration, given its four evaluation and iteration phases used to construct a validated artifact. The various iterative steps of evaluation and construction/refinement of Design Science Research, therefore, guide the research process to ensure relevance and rigour of the design principles [47-51].
In the first phase a literature review was conducted using context analysis to deter-
mine the relevance of the field of study. The initial design and development of a con-
ceptual framework were based on insights gained from the literature study. In the sec-
ond phase the aim was the formative evaluation of the initial artifact design (ex ante
evaluation), resulting in a refined artifact. During the third phase, which is the main
focus of this paper, the second version of the framework was validated, and the results
of this ex post evaluation were used to further refine the artifact before it was validated
and improved in a final phase.

The research was conducted at Midrand Graduate Institute (MGI), a Private Higher
Education Institution (PHEI) in South Africa, which changed its name in 2016 to Pear-
son Institute of Higher Education. The targeted participant population was moderators
selected from all the faculties in the institution. Participants were formally approached
via email to take part in the research. No incentives were offered to any participant and
participation was voluntary. Although all 75 of the moderators of the institution (MGI)
were invited to take part in the research, only 30 of the eModerators participated in the
survey (phase two). During the third phase two moderators per faculty were selected
and invited to be interviewed. Only six out of the 30 eModerators volunteered to be
interviewed during phase three. English was used as language to conduct the interviews
and each interview lasted on average 20 to 30 minutes. The participants were not close
to the researcher and for that reason interviews were conducted using Skype if face-to-
face interviews could not be conducted.

A case study strategy, which concerns the interpretation of qualitative data collected,
was used to generate data. An interpretive philosophical worldview was used because
it allows contemplative reflection on the participants’ views. The data generation meth-
ods used during the third phase include interviews with eModerators from different
faculties. Interviews were designed using open-ended questions.

The evaluation criteria used to evaluate the artifact were selected after thorough con-
sideration of various aspects. For example, the evaluation criteria had to be independent
of the artifact type [52]. Hevner et al. [47] also indicate that artifacts should be evalu-
ated using criteria relevant to the requirements of the context in which the artifact is
implemented. Participants had to evaluate the framework using the following evalua-
tion criteria [47, 53]: simplicity, generality, exactness, clarity, completeness and rele-
vance of the three levels (Environmental, eModeration requirements and eModeration
User Experience constructs levels).

Prior to the interview a diagrammatical representation of the framework, as well as
a detailed explanation of the constructs, was emailed to the eModerators. The inter-
views were transcribed by an independent person. Interviews were conducted using the
telephone and Skype as tools. Notes were also made during the interview process. To
verify the correctness of data the captured responses were emailed back to the inter-
viewees for confirmation and clarification to determine if the researchers understood
the answers to questions correctly, using thematic analysis [54]. The comments and
responses from the eModerators were coded into a structure in which themes emerged
across the subset of data. Common themes were defined, grouped together and named
accordingly. The comments, feedback and responses from the participants (eModera-
tors) were used to refine the User Experience Evaluation Framework for eModeration.
# 4 Results and findings

As mentioned by Mahlke and Thüring [33], user experience evaluations should be based on non-instrumental qualities, effect and emotion. In this paper the findings were reached by using principles of user experience heuristics identified by Hassenzahl and Monk [55], as well as Väänänen-Vainio-Mattila and Wäljas [35].

Based on the questions related to the **adequacy and appropriateness**, of the three identified levels of the framework, the eModerators agreed that there is no need to add an additional level. One participant (D) indicated that, if an extra level were to be added, it would make the framework to complex and confusing, while Participant F commented: “Would not add another level as the three levels cover all aspects of eModeration”. Only Participant E highlighted a specific theme namely ‘context’. The participant found it difficult to understand how and where the construct ‘context’ should appear in the framework. The comment was taken into consideration with the refinement of the framework and more details were added to the evaluation criteria associated with ‘context’. The ‘context’ construct is included under the user experience level and includes “characteristics of module assessment” as requested by Participant E.

**Under the environmental level,** eModerators were asked if the framework could be used in other organisations besides PHEIs. Organisations are a construct under the environmental level. According to Participant A, who is also an expert in eLearning systems, “...the criteria could also be applied to any system as both a quality assurance and user experience evaluation framework”. Participant D indicated that “[p]ublic HEIs... colleges, schools and any other academic institution[s] having access to internet might also benefit, especially where external moderators are needed”, and Participant E agreed. Participant C, however, was of the following opinion: “No, I think it is customised for Higher Education Institutions only”. Participant F identified the possibility of using the framework not only for moderation purposes, but also in instances where documents require quality assurance. As a result, it was decided to use “Higher Education Institutions” in general instead of differentiating between Public and Private HEIs to facilitate the inclusion of colleges, schools and other academic institutions. Embedded within the ‘user’ construct are roles and responsibilities. Participant C expressed the following opinion: “Within the environment level, my opinion is that constitutional regulations and policies might also influence this level. I am not sure, however, if such regulations exist and are prescribed and enforced by the Department of Higher Education”. Academic institutions in South Africa, and specifically HEIs, are governed by the country’s Department of Higher Education, which prescribes and requires HEIs to externally moderate all exit level modules [56]. However, each HEI has its own set of policies and procedures incorporating regulatory requirements. Instead of covering policies under the environmental level, it was included and incorporated under the eModeration requirements level. To conclude, two themes emerged under the environmental level: the framework could be used in more than one type of academic institution and it could be used for other purposes, not only for moderation of examination scripts. It was agreed by the participating eModerators that the environmental level was appropriate with respect to simplicity, generality, clarity and relevance.
**Under the eModeration requirements level**, IT support emerged as an additional theme. IT support includes hardware, software and people. The roles and responsibilities of IT support were included under ‘users’, but the tasks associated with provision of IT support were included as an additional construct during the refinement of the framework. Participant A recommended that, if an institution cannot afford it to appoint an individual to take the full responsibility of ‘IT support’, these roles be assigned to the eModeration systems operator. Participant C also recommended that “system maintenance and upgrades” be added under support.

Participants were of opinion that the evaluation criteria discussed under the eMod-
eration requirements level were clearly explained and comprehensive. A theme that
emerged under the eModeration requirements level relates to the distribution of login
details to be sent automatically via email to eModerators instead of a developer email-
ing the manager to communicate the details to them. Participants considered the eMod-
eration requirements level to be adequate with respect to exactness, simplicity, compre-
hensiveness and relevance. Participant D commended the uploading process: “The up-
load process works smoothly; I was quite amazed by the effectiveness.” It is thus im-
portant to note that, when evaluating an eModerate system, users should make sure that
the system makes provision for users to upload documents easily and that the process
is effective.

**The eModeration user experience construct level** was divided into two categories: instrumental and non-instrumental qualities [32, 34, 57]. The participants agreed that the constructs were relevant, clearly explained, complete and comprehensive enough for the eModeration user experience level. Participant C indicated: “I would add an item such as system maintenance under the heading of error prevention”, which was consequently added under the element of system maintenance during the refinement of the artifact. Overall, the participants agreed that the framework presents a relevant solution to a very practical problem, as indicated: “Perfect for external moderation! Very useful for internal moderation as well to keep track and record of each semester’s examination results and moderation”. Fig. 1 represents the revised User Experience Evaluation Framework for eModeration after refinement based on feedback received during interviews with eModerators.
<figure><img src="image_2.png" /><figcaption>Fig. 1. User Experience Evaluation Framework for eModeration</figcaption></figure>

The eModerators were satisfied with the instrumental and non-instrumental qualities identified in the artifact. The third evaluation phase of the User Experience Evaluation Framework for eModeration provided useful feedback to refine the artifact. IT support was added to the conceptual framework as extra user roles and responsibilities with a separate IT support element under the eModeration requirements level. In conclusion, the interviews with eModerators provided useful feedback to incorporate into the refinement of the User Experience Evaluation Framework for eModeration before it was evaluated in the last phase to validate the artifact with an external institution (the results of which fall outside the scope of this paper). A more detailed discussion on the eModeration Framework can be found in Van Staden [58].

# 5 Conclusion

This paper makes a theoretical contribution by responding to the need for research on user experience and by presenting an evidence based artifact to guide and evaluate the implementation of an eModeration system. The proposed User Experience Evaluation Framework for eModeration is the artifact that expands existing theoretical framework to describe eModeration in the field of higher education. The User Experience Evaluation Framework for eModeration also contributes to the solution of a local relevant problem in a developing country, specifically with respect to improving resilience as well as providing an e-Service to educators. This framework supports all the primary foundations of resilience, namely robustness, self-organisation and learning. This framework can also guide the transformation from manual moderation to electronic moderation, with the focus on the use of ICT as a productive tool (eModerate system) used by eModerators and management to fit their objective to produce digital content
(electronic examination scripts). The paper explained how e-Services can be used by educators to improve human experience of electronic moderation with an emphasis on improving the quality of educational assessment practices in developing countries. The authors trust that the proposed framework could be used to facilitate and improve quality control in tertiary assessment practices, while contributing on a theoretical level to Information Systems knowledge regarding human-computer interaction and user experience in eModeration.

# 6 Acknowledgements

This work is based on the research supported by the South African Research Chairs Initiative of the Department of Science and Technology and National Research Foundation of South Africa (Grant No 98564).

# References

1. Van Staden, C.J. IT moderation going green! In: Proceedings of SAICSIT'10 Conference, pp. 1-3. UNISA Production Printers, Bela Bela, Limpopo, South Africa (2010). DOI:10.1145/1899503.1899559

2. Heeks, R. Ospina, A.V.: Conceptualising the link between Information Systems and resilience: A developing country field study. Information Systems J. 29(1), 70-96 (2018).

3. Dennick, R., Wilkonson, S., Purcell, N.: Online eAssessment: AMEE Guide No. 39. J. Medical Teacher 31(3), 192-206 (2009).

4. Guàrdia, L., Crisp, G., Alsina, I.: Trends and challenges of e-assessment to enhance student learning in higher education. In: Cano, E., Ion, G. Innovative Practices for Higher Education Assessment and Measurement, pp. 295-415. IGI Global Hershey PA, USA (2017).

5. Whitelock, D.: Activating assessment for learning: Are we on the way with Web 2.0? In: Lee, M.J.W., McLoughlin, C. Web2.0-Based e-learning: Applying Social Informatics for Tertiary Teaching. Information Science Reference, pp. 319-342. IGI Global, Hershey, PA 17033 The Open University, UK (2010).

6. Salmon, G.: E-moderating: The key to teaching and learning online. Kogan, Page, London (2003).

7. Salmon, G.: E-tivities: The key to active online learning. Taylor & Francis Group, New York (2013).

8. Greatorex, J.: Moderated e-portfolio evaluation. Evaluation and Validation Assessment Directorate, UCLES (2004).

9. Bridge, P., Appleyard, R.: A comparison of electronic and paper based assignment submission and feedback. British Journal of Education Technology 39(4), 644-650 (2008). DOI: 10.1111/j.1467-8535.2007.00753X

10. Morgan, A.: eModeration: Contextualising online learning in undergraduate nurse education. Asian Journal of Nursing, 11(1), 48-53 (2008).

11. Vlachopoulos, P.: The nature of e-moderation in online learning environments. In: Proceedings of LICK 2008 Symposium, pp. 48-57. Napier University TESEP. Edinburg http://www.napier.ac.uk/transform (2008).
12. Wichmann, A., Giemza, A., Hoppe, U., Collide-Group.: Effects of awareness support on moderating multiple parallel e-discussions. Computer Supported Collaborative Learning Practices CSCL2009. ISLS. pp 646-650 (2009).

13. Adie, L.: An investigation into online moderation. Assessment Matters 3, 5-27 (2011).

14. Adie, L.: Towards a theoretical framework for online professional discussions. Journal of Learning Design JLD 7(3), 54-66 (2014).

15. Grainger, P., Adie, L., Weir, K.: Quality assurance of assessment and moderation discourses involving sessional staff. Journal Assessment & Evaluation in Higher Education 41(4) 548-559 (2016). http://doi.org/10.1080/02602938.2015.1030333

16. Adie, L.: Changing assessment practices: The case for online moderation. In: AARE 2008 Proceedings of International Educational Research Conference, pp. 1-14. Australian Association of Research in Education, AARE, Brisbane (2009).

17. Adie, L., Lloyd, M., Beutel, D.: Identifying discourses of moderation in higher education. Journal of Assessment & Evaluation in Higher Education 38(8), 968-977 (2013).

18. MGI.: Assessment Policy. Midrand (2010).

19. Van Staden, C.J., Van Biljon, J.A., and Kroeze, J.H.: Adopting eModeration: Understanding the user experience in the organisation. 8th European Conference on IS Management and Evaluation. ECIME 2014 Conference, pp. 356-365. Ghent, Belgium (2014).

20. Van Staden C.J., Van Biljon, J.A. and Kroeze, J.H.: eModeration: Towards a user experience evaluation framework. In: SAICSIT 2015 Proceedings of the 2015 Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists. ICPS ACM. Article 39. Stellenbosch, Western Cape, South Africa (2015). DOI: http://dx.doi.org/10.1145/2815782.2815821

21. Preece, J., Sharp, H., Roger, Y.: Interaction design. 3rd edn. John Wiley & Sons Ltd, Chichester, West Sussex, United Kingdom (2011).

22. ISODIS9241-210.: Ergonomics of human systems interaction – Part 210: Human-centred design for interactive systems (formerly known as 13407). International Organisation of Standards (2010).

23. Hassenzahl, M., Tractinsky, N.: User experience – a research agenda. Behaviour and Information Technology 25(2), 91-97 (2006). DOI: 10.1080/01449290500330331

24. Van Schaik, P., Van Aranyi, G.: Research - white paper: Model-based User-Experience. Teesside University (2014).

25. Law, E., Kort, J., Roto, V., Hassenzahl, M., Vermeeren, A.P.O.S.: Towards a shared definition of user experience. In: Proceedings of Computer-Human Interaction CHI 2008 Special Interest Group, pp. 2395-2398. Association for Computing Machinery ACM, Florence, Italy (2008).

26. Law, E., Roto, V., Hassenzahl, M., Vermeeren, A., Kort, J.: Understanding, scoping and defining user experience: A survey approach. In: Proceedings Computer Human Interaction CHI'09 User Experience, pp. 719-728. Association for Computing Machinery ACM Special Interest Group for Computer-Human Interaction SIGCHI, Boston, MA, USA (2009).

27. Law, E.: The measurability and predictability of user experience. In: Conference Proceedings of EICS'11, pp. 1-10, Pisa, ACM New York, USA, Pisa, Italy (2011).

28. Roto, V.: Web browsing on mobile phones – characteristics of user experience. Doctoral Dissertation. Helsinki: University of Technology (2006).

29. Van Der Peijl, J., Klein, J., Grass, C., Freudenthal, A.: Design for risk control: The role of usability engineering in the management of use-related risks. Journal of Biomedical Informatics 45, 795-812 (2012).
30. Vermeeren, A.P.O.S., Law, E., Roto, V., Obirist, M., Hoonhout, J., Väänänen-Vainio-Mattila, K.: User experience evaluation methods: current state and development needs. In: Proceedings of the Workshop on Recommendation and Personalization in Electronic Commerce, International Conference on Adaptive Hypermedia and Adaptive Web Based Systems. pp. 530-531. Reykjavik, Ireland (2010).

31. McCarthy, J., Wright, J.: Technology as experience. MIT Press, Massachusetts (2007). DOI: 10.1145/1015530.1015549

32. Väätäjä, H., Koponen, T. Roto, V.: Developing practical tools for user experience evaluation: A case from mobile news journalism. In: Proceedings of ECCE'09 European Conference on Cognitive Ergonomics beyond the Product – Understanding Activity and User Experience in Ubiquitous Environments, article 23, ACM, Helsinki, Finland (2009).

33. Mahlke, S., Thüring, M.: Studying antecedents of emotional experiences in interactive contexts. In: Proceedings of CHI 2007, vol. 1, pp. 915-918. Conference on Human Factors in Computing Systems, San Jose: CHI (2007).

34. Hassenzahl, M.: User experience (UX): Towards an experiential perspective on product quality. In: Proceedings of the 20th International Conference of the Association Francophone d'Interaction Homme, pp. 11-15. Machine Metz, Proceedings of IHM (2008).

35. Väänänen-Vainio-Mattila, K., Wäljas, M.: Development of evaluation heuristics for web service user experience. In: Proceedings of Computer-Human Interaction CHI EA 2009, 27th International Conference on Extended Abstracts on Human Factors in Computing Systems, pp. 3679-3684. ACM, Boston, MA, USA (2009).

36. Nielsen-Norman-Group, J.: User satisfaction vs. performance metrics. http://www.ngroup.com/articles/satisfaction-vs-performance-metrics/ (2012).

37. Tullis, T. Albert, B.: Measuring the user experience: Collecting, analyzing and presenting usability metrics. 2nd edn. Morgan Kaufman Elsevier, Waltham (2013).

38. Abdelnour-Nocera, J. Densmore, M.: A review of perspectives and challenges for international development in information and communication technologies. Annals of the International Communication Association. Routledge, Taylor & Francis (2017).

39. Marais, M.A.: ICT4D and sustainability. In: Eng, P.H., Mansell, R. (eds.) The International Encyclopedia of digital Communication and society 1. Major Reference works (2015).

40. Heeks, R.: ICT4D 2.0: The next phase of applying ICT for international development. Computer 41(6), 26-33 (2008).

41. Heeks, Richard. Information and communication technology for development (ICT4D). Routledge, (2017).

42. Paluch, K.: User experience design blog: Commentary on strategy and design of interactive products. http://www.montparnas.com/articles/what-is-user-experience-design/ (2006).

43. Rubinoff, R.: How to quantify the user experience. Chicago: APA (2009).

44. Van Staden, C.J., Van Biljon, J.A. and Kroeze, J.H.: Using a user experience evaluation framework for eModeration. In: Olugbara, O., Millham, R., Heukleman, D. (eds.) Proceedings of IEEE ICTAS 2017, pp. 15-21. Durban, South Africa (2017).

45. Schulze, K., Krömker, H.A.: A framework to measure user experience of interactive online products. In: Proceedings of MB'10 Conference, article 14. Association for Computing Machinery ACM, New York, Eindhoven, Netherlands (2010). DOI: 10.1145/193144.1931358

46. Ouma, S.: M-health user experience framework for the public healthcare sector. PhD Dissertation, Nelson Mandela Metropolitan University NMMU (2013).

47. Hevner, A.R., March, S.T., Ram, S.: Design science in Information Systems research. MIS Quarterly 28(1), 75-105 (2004).

48. Hevner, A.R.: A three cycle view of Design Science Research. Scandinavian Journal of Information Systems 19(2), 87-92 (2007).
49. Sonnenberg, C., Vom Brocke, J.: Evaluations in the science of the artificial – reconsidering the build-evaluate pattern in Design Science Research. Design Science Research in Information Systems: Advances in Theory and Practices 7286, 381-397 (2012). http://link.springer.com/chapter/10.1007/978-3-642-29863-9_28

50. Gregor, S.: The nature or theory in Information Systems. MIS Quarterly 30, 611-642 (2006).

51. Gregor, S., Jones, D.: The anatomy of a design theory. Journal of the Association of Information Systems 8(5), 312-335 (2007).

52. Aier, S., Fischer, C.: Criteria for progress of Information Systems design theories. Information Systems and E-Business Management 9(1), 133-172 (2011).

53. Rossmann, M., Vessey, I.: Towards improving the relevance of Information Systems research to practice: The role of applicability checks. MIS Quarterly 32(1), 1-22 (2008).

54. Braun, V., Clarke, V.: Using thematical analysis in psychology. Qualitative Research in Psychology 3(2), 77-101 (2006). http://dx.doi.org/10.1191/1478088706qp063oa

55. Hassenzahl, M., Monk, A.: The inference of perceived usability from beauty. Human-Computer Interaction 25(3), 235-260 (2010).

56. SAQA: Criteria and guidelines for assessment of NQF registered unit standards and qualifications. SAQA, Pretoria (2010).

57. Van Staden, C.J., Van Biljon, J.A. and Kroeze, J.H. eModeration: The validation of a user experience evaluation framework. In: Wu, T-T., Huang, Y-M., Shadiev, R, Lin, L., Istenič Starčič, A. Proceedings of Innovative Technologies and Learning First International Conference, ICITL 2018, LNCS, pp. 241-252. Portoroz, Slovenia (2018).

58. Van Staden, C.J.: User experience evaluation of electronic moderation systems: A case study at a private higher education institution in South Africa. Doctoral Dissertation, University of South Africa, School of Computing (2017).