# Requirements for an eModeration System in Private Schools in South Africa

Vanitha Rajamany, Corné J. Staden, Judy Biljon

► To cite this version:

Vanitha Rajamany, Corné J. Staden, Judy Biljon. Requirements for an eModeration System in Private Schools in South Africa. 19th Conference on e-Business, e-Services and e-Society (I3E), Apr 2020, Skukuza, South Africa. pp.557-568, 10.1007/978-3-030-44999-5_46. http-03222835

HAL Id: hal-03222835

https://inria.hal.science/hal-03222835

Submitted on 10 May 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Requirements for an eModeration system in private
schools in South Africa

Vanitha Rajamany¹ [0000-0002-8210-377X] C.J. van Staden² [0000-0002-3991-3128], J.A. van Biljon³ [0000-0002-4646-1641]

School of Computing, UNISA, Pretoria

¹ 7232969@mylife.unisa.ac.za, ² vstadcj1@unisa.ac.za, ³
vbiljja@unisa.ac.za

**Abstract.** Despite the growing importance of digitization in all aspects of teaching and learning, digital moderation (eModeration) has received little attention in research or practice. Considering the secondary school environment, no evidence-based requirements could be found for the development of a digital moderation system. This finding provides the rationale for this investigation into the requirements for an efficient eModeration system for IT and CAT assessments at grade 12 level in South Africa (SA). This study draws on eModeration literature as well as technology adoption literature to identify a set of pre-adoption eModeration requirements against the background of the challenges and benefits of eModeration. The research design involved a single exploratory case study with IT and CAT moderators as units of analysis. Qualitative data analysis using Atlas.ti V8 was conducted on the 61 responses received from an online survey and 4 responses received from a focus group interview. The findings confirmed the absence of customized eModeration systems. The main contribution is the eModeration requirements identified from literature and then refined by triangulation with the empirical findings of the survey. The research has practical value in guiding the design of eModeration systems for the school environment.

**Keywords:** eModeration, quality assurance, paper-based moderation, technology acceptance, annotation tools, security.

# 1 Introduction

eModeration can be explained as the use of electronic media to support moderation and the secure storage of student evidence; quality assurance evidence of assessment; record results electronically and upload students' work to a secure site for moderation [1]. Van Staden [2] describes eModeration as an electronic quality assurance process using an online tool. The New Zealand Qualifications' Authority established the Digital Moderation Project for completing all external moderation online [3]. The objective was to submit assessments via an online digital platform by providing an application to support moderation practices [4]. However, there is no evidence of the successful implementation nor the efficacy of such a system within the New Zealand educational system.
In the SA context, Van Staden et. al., [5] demonstrated the efficacy of an eModera-
tion system which was successfully implemented at a higher education institution.
However, despite the identified benefits of optimizing moderation procedures, reducing
the time taken for moderation processes and decreased costs, the application of online
moderation in the secondary school environment is limited. Paper-based moderation is
still extensively used at academic institutions in SA [6].

Drawing on the existing work of Van Staden [6] and reports commissioned by the
New Zealand Qualifications Authority [7], eModeration is defined as the electronic
quality assurance of portfolio tasks by external moderators within a digital environ-
ment. This study draws on prevailing eModeration and technology adoption literature
and empirical research in the private secondary school environment in SA to provide
an evidence-based contribution to the requirements for an efficient eModeration sys-
tem.

This study is structured as follows: an introduction to the study, a preliminary liter-
ature review of the requirements for an eModeration system and introduction to four
existing technology acceptance models whose constructs formed the basis for this
study, an overview of the data collection methods and the findings, an analysis of the
qualitative data and the final section concludes this paper.

## 2 Preliminary Literature Review

A preliminary literature review of the benefits, challenges and requirements from the literature are now presented.

2.1 Benefits and challenges of eModeration

The benefits of eModeration are directly related to the disadvantages of paper-based
moderation. The latter is less efficient, both in terms of time and resources. For exam-
ple, the resources required for transporting assessments to moderators and the space for
archiving the scripts. Additionally, online moderation limits the loss of moderation re-
ports and assessments (which are conventionally couriered [2; 9]), in transit.

Challenges and benefits identified from the literature are discussed in the context of
participant responses in section 3.3. The challenges relate mostly to the usability of the
systems, e.g. judgements are impaired when moderators must manage different pages
in documents [10]. Searching through different portfolios presents difficulties when
viewing and comparing different candidates’ work. Studies indicate that an annotation
tool is invaluable in developing shared perspectives of standards in digital moderation
[11; 12]. The limited accessibility of such tools increases cognitive demands on the
educator and the process can become less reliable [11]. Additionally, teachers involved
in eModeration must be confident in the use of the technology [1].
## 2.2 Requirements for an eModeration System

Technological developments offer the possibility of developing online tools for remote standardization of marking [13]. The following requirements for an eModeration system have been identified from literature [6; 14]:

* Reusability: existing content can be reused on different platforms.

* Manageability: refers to the ability of a system to track moderation procedures.

* Accessibility: stakeholders can access content without time or place constraints.

* Durability: unnecessary to redevelop content when new versions are installed.

* Scalability: minimal effort is expended to extend the system.

* Affordability: the system must be cost effective for the primary users of the system or based on free technologies so that the institution has no licensing issues.

* Security and reliability: each user should have her own space with email and discussion facilities. The privacy and confidentiality of each school's work must be guaranteed while all access to a student's private space should be denied.

* Usability: the tool should be user friendly and self-explanatory.

* Portability: possible to set up the system on another server without difficulty.

* Infrastructure: hosting institutions should have adequate infrastructure.

* Bandwidth: increased bandwidth will increase the efficiency of the system.

An important consideration is if the technology enhances task performance and if it will provide an e-Service to potential users. If there is a good fit between the task and the technology being used to complete the task, then there is increased probability of greater utilization which is important for online standardization. Online standardization has distinct characteristics and best results can only be obtained if the requisite processes are supported by the technology [15].

## 2.3 Technology Acceptance Models

Studies into the acceptance of technology stem from the sphere of information systems (IS). Various technology acceptance models for instance, Technology Acceptance Model (TAM) [16]–[18], Task-Technology-Fit (TTF) [17], Technological-Organizational-Environmental (TOE) [19] and the Human, Organization and Technology (HOT-Fit) model [20] exist in IS literature. A challenge is that technology acceptance models relate to existing system use while the context for this study is the pre-adoption of an eModeration system. The basic premise of these models is summarized in **Table 1**.

See Rajamany, Van Staden and Van Biljon [26] for a detailed comparison of the pre-adoption constructs of these models, namely TAM, TTF, TOE and HOTFIT. A table (cf. https://cutt.ly/NetWOjq) was created to determine the commonality of the constructs amongst the models as well as commonality with constructs mentioned by respondents to the survey and comments made by focus group participants. The constructs were prioritized by centrality, i.e. those occurring across all three data streams formed the basis for the extraction of final requirements for an eModeration system.
<table><caption>**Table 1.** Summarized Theoretical models</caption><thead><tr><td>Model</td><td>Fit between human and technology</td></tr></thead><tbody><tr><td>TAM</td><td>users will not readily accept a system, irrespective of how efficient the system is, if they cannot immediately identify tangible benefits in their job performance as a direct result of utilizing the system [23].</td></tr><tr><td>TTF</td><td>an IT system will be used only if the functionality supports the user's activities. Any system not offering sufficient support will therefore, by extension, not be utilized [24].</td></tr><tr><td>TOE</td><td>technological, environmental and organizational factors influence ICT adoption [19].</td></tr><tr><td>HOT-Fit</td><td>the user, technologies and organizational contexts cumulatively affect the net benefits of using an IS. Human, organizational and technical elements should have a mutual alignment to ensure successful implementation [25].</td></tr></tbody></table>

## 3 Data collection and findings

The pragmatist philosophy views research as a human experience based on the beliefs of individual researchers [27]. Pragmatism proposes a philosophy extending beyond “what works” [28] and encourages an inquiry based process of searching for knowledge with the aim of solving societal problems. Pragmatism is particularly suitable for the current study as it is envisaged that the resulting body of knowledge will inform future practices and policies within schools and assessment bodies.

The literature review informed the design of a customized questionnaire developed using Google forms and distributed via email to teachers at private schools in SA. The online survey captured qualitative and quantitative data. The responses to the open-ended questions (qualitative) provided a basis for the structuring of the questions for a focus group interview conducted with 4 moderators.

Focus group interviews explore the attitudes and perceptions of a homogeneous group of people focused on a given issue [29] and use a purposeful sample composed of information-rich participants resulting in a greater sharing of insights [27; 28]. The demographics of the survey population is depicted in **Table 2**. The survey was distributed to 122 IT and 145 CAT teachers in private schools via email; 61 of whom responded. Focus group participants (N = 4) were moderators who volunteered to participate after being contacted via email.

<table><caption>**Table 2.** Participant Demographics</caption><thead><tr><td>Role (N= 61)</td><td>%</td></tr></thead><tbody><tr><td>IT Educator</td><td>53,20%</td></tr><tr><td>CAT educator</td><td>24,20%</td></tr><tr><td>CAT and IT educator</td><td>6,4%</td></tr><tr><td>Cluster moderator</td><td>3,30%</td></tr><tr><td>Regional moderator</td><td>4,90%</td></tr><tr><td>National moderator</td><td>6,60%</td></tr><tr><td>Other</td><td>1,40%</td></tr></tbody></table>

<table><thead><tr><td>Technology Experience (N = 61)</td><td>%</td></tr></thead><tbody><tr><td>&lt; 1 year</td><td>3,30%</td></tr><tr><td>1 - 5 years</td><td>8,30%</td></tr><tr><td>6 - 10 years</td><td>8,30%</td></tr><tr><td>&gt; 10 years</td><td>80%</td></tr></tbody></table>
Both qualitative and quantitative data was collected, but due to space constraints only the qualitative data is reported on. The survey contained open-ended questions requiring participants to express their opinions on the challenges and benefits of current moderation practices and the perceived challenges, benefits and requirements of an eModeration system. The data from the survey and the focus group was analyzed thematically where a theme is considered to be a coherent and meaningful pattern in the data which captures an important aspect about the data in relation to the research question [31].

Section 3.1 presents the findings from the survey, section 3.2 presents the findings from the focus group and section 3.3 presents an integrated analysis of the findings.

## 3.1 Qualitative findings from survey

Participants' comments around the challenges and benefits of current moderation practices and the perceived challenges, benefits and requirements for an eModeration system were imported into Atlas.ti V8. Words which appeared most frequently were used to identify themes around which the discussion of the findings is structured.

- **Challenges of current moderation practices:** The greatest challenge is that it is time consuming to scan portfolios before uploading these documents digitally. Huge amounts of paper is wasted when moderating manually. A related challenge of paper-based moderation is that of "inconsistent record-keeping" and the fear of losing evidence of moderation. Together with the costs of paper, printing and other consumables, costs are incurred when portfolios are couriered to "out of town" schools.

- **Benefits of current moderation practices:** Participants' responses were mostly positive for those who use some form of digital moderation. The most frequent response was that Google Drive or Google docs are used for moderation processes. Respondents agreed that it is easy and effortless to use Google Drive. Feedback is timely and eModeration is especially convenient when the school is in a rural area. A related benefit is that eModeration reduces the costs of transporting or posting portfolios.

- **Challenges of eModeration:** The most common challenge was Internet connectivity. Additional challenges: differing file formats, access to necessary resources like scanners, a knowledge of the system, versioning problems, the time required to learn to use the system and the time consumed in scanning the necessary documents. A respondent mentioned "Annotation of content." However, it is not evident if the reference was to the availability of the tools or the knowledge required to use these tools. A significant finding is that although some respondents indicated using a digital moderation process as evidenced by the following: "I have been using an eModeration system for the past 3 years and will never go back to a paper-based system again", other respondents responded as: "I have only used Google Docs and am Will (sic) be honest to say I do not know what an eModeration system is."

- **Perceived benefits of an eModeration system:** ease of use, faster feedback, cheaper than couriering portfolios, faster to find digital files, easy to upload files.
- **Requirements for an eModeration system:** Respondents mentioned Internet bandwidth, multiple access and reliability but did not expand on these aspects. The requirements for an eModeration system are summarized using the specific quotations of respondents:

* Connection availability: "A good internet connection and a secure drop site."

* Connection speed: "Fast connection speed."

* Additional resources: "Equipment. Good quality scanner which can handle bulk scanning."

* Attitude: "A positive attitude towards the use of this technology."

* Availability of system: "Being always available."

* Annotation: "Annotation that will allow for different moderation styles."

* Communication: "It should make communication easier."

* Ease of use: "An easy to use interface with standardized organization for files and folders."

* Flexibility: "Flexibility so one can moderate one section only or start at the back."

* Responsiveness: "The system needs to be responsive and not have huge amounts of latency."

* Security: "There should be sufficient security to ensure that the portfolios can only be seen by the school and relevant moderator."

* Timely feedback: "It should save me time. It should also give you feedback as to whether the other person is actively involved in the moderation process."

## 3.2 Focus Group Findings

The focus group interview was structured around themes that emerged from the responses to the open-ended questions in the questionnaire. Participants' responses were loaded into QDA Miner Lite and the various subthemes were analyzed. The significant findings for each theme are summarized as:

- **Challenges of current moderation practices:** turn-around time; more difficult to keep track of hard copies; physical waste of resources; security; slow and inflexible.

- **Challenges of manual moderation:** easier to lose hard copies; physical waste of resources; many drafts must be done on paper.

- **Challenges of eModeration:** resistance to change; competence; file formats; document formatting; time taken for feedback; slow to download files; need for many screens.

- **eModeration vs Manual Moderation:** easier to track, view and review changes on a digital version; eModeration takes longer than manual moderation.

- **Benefits of eModeration:** easier to contact the moderator.

- **Requirements for an efficient eModeration system:** more than one screen; multiple views; revision histories; easy to use and advanced annotation software; voice over to leave a comment; a call button for communication.

While the consensus was that scanning of documents is time consuming, participants acknowledged that "*the pen and paper connection*" is important. However, the issue of
security pertaining to written assessments cannot be trivialized. Focus group partici-
pants agreed that security is the largest challenge posed by paper-based moderation. A
qualitative analysis of the data indicates that the frequency at which security was men-
tioned in the context of paper-based moderation is 30%.

3.3 INTEGRATED DATA ANALYSIS

*In this section, the challenges, benefits and requirements are synthesized from the literature and data obtained from the survey and focus group interview.*

— Challenges .

Cost: An electronic moderation system requires additional technology like scanners thus increasing the initial costs [6].

*Infrastructure:* The ICT infrastructure necessary to develop digital portfolios is a particular challenge [12], [32]. A good computer network capable of high-speed uploading and downloading of projects is essential. Instability in the network can create challenges for uploading projects. Having appropriate access to the schools’ network systems is also important [1]. Focus group interviewees agree that the Internet connection hampers the process by slowing down the downloading of files as evidenced by participant B’s response: “*the frustration is waiting for the assessment to load which slows down the process.*”

*Technology and Availability of Resources:* technological limitations can hamper the implementation of eModeration systems [33]. Focus group participants concur, indicating: “*At this stage technology is failing the process.*” Although all participants are from private schools, the assumption that all schools are equally resourced cannot be made. Hence available resources at individual schools play a significant role in the efficacy of eModeration. It is therefore reasonable to assume that the lack of adequate resources could be a limiting factor in digital moderation.

*Training:* Making comparative and holistic judgements of students’ work is a critical aspect of moderation [33]. Participants A and B stated that “*Having to highlight on a pdf or add a comment box to annotate a pdf requires the person moderating to be trained on the technology.*” Additionally, participant C indicated that “*A challenge is that people are resistant to being trained on a new technology.*” These statements confirm literature findings that assessors’ approach to online standardization may be hampered by discomfort in using technology and the skills required to interact with an eModeration system [8; 34]. Contrary to literature findings, none of the survey respondents indicated discomfort in using technology. A factor in the contradiction between the findings of this study and the literature reviewed is that respondents were teachers whose professions require prowess in using digital resources. Whereas the literature generalized the challenges to all educators, the survey and focus group in this study isolated IT and CAT educators. It is thus not surprising that these teachers would either be proficient or possess the motivation to learn how to use unfamiliar technology.

*Confidentiality and reliability:* Participants concurred with Van Staden [9] that phys-
ical copies can easily be misplaced leading to issues around confidentiality. If one mis-
places a learner’s work, then there is no evidence of submission which creates the added
problem of reliability. With eModeration, proof of moderation is easily available with
tracked changes.

*Time:* Turnaround time is a challenge of paper-based moderation. Many drafts must be done; involving rewriting information. Data from the focus group and the survey confirms Van Staden’s [9] findings that moderators prefer to use an eModeration system. For instance, participant A indicated: “tending towards digital because of poor security with paper” while participant B indicated that “digital moderation is preferable because there is an audit trail” and a survey respondent stated, “I prefer sending a digital portfolio for moderation that (sic) making endless copies for each student file.”

Contrary to the assertion that electronic development and storage of evidence results
in greater flexibility, convenience and accessibility [34], participant A indicated that
the “loss of flexibility is huge with digital moderation.” However, upon further interro-
gation it was determined that this view was being expressed on current moderation
practices as evidenced by: “the way in which it is currently being done is slow and
inflexible.” This result is not surprising given that there is no standard, usable dedicated
eModeration system in place.

— Benefits of eModeration

Summarized benefits are: typing comments is faster; easy to create a reusable bank of comments reducing the amount of work and time taken; better quality; eliminates the possibility of mistakes; easy to determine if changes have been made. Feedback is faster as evident by the comment “*reduces the time of couriering moderation documents and having to wait for the results to be delivered. With digital moderation, as soon as the submit button is clicked, all parties concerned receive notification.*”

— Requirements of an eModeration System

Based on the results from the survey and focus group interview, the perceived require-
ments for an eModeration system can be summarized as: need for multiple views, easy
to use annotation tools, effective communication, security, availability and flexibility.
Responses from the focus group strongly confirm the literature findings; with partici-
pants mentioning multiple views and having “*more than one screen*” so that “*all rele-
vant documents can be viewed side by side*” with great frequency. Additionally, partic-
ipants indicated that “*these views should also show revision histories*” and that it would
be useful to have “*one document with all changes made right from the first version.*”

Consistent with the findings discussed in 2.1, focus group participants indicated that an eModeration system should include easy to use “*top tier*” annotation software; a sentiment echoed by respondents of the survey. For instance: “*Annotation has to be quick and simple to use.*”

The privacy and confidentiality of each school’s work must be guaranteed while all
access to a student’s private space should be denied [6]. This important theme of secu-
rity recurs in the data extracted from additional comments provided by respondents to
the survey. The statements “*Security - papers may not leak.*” and “*Security for the doc-
uments using a per session key of suitable length that changes for each session*” are
representative of statements made by these respondents.
Contrary to the results from the literature reviewed and the survey responses, and despite security being a recurrent theme during the discussion on manual moderation, it was surprising that security was not overtly mentioned when participants were asked to elaborate on the requirements for an eModeration system. A possible explanation for this contradiction is that while security is a huge issue from the organization's perspective; it is not necessarily an issue that directly impacts users of the system. The focus seems to be on convenience and flexibility of use rather than on essential aspects. It seems reasonable therefore to conclude that users focus on aspects that affect them directly as evidenced by the frequency of hardware requirements being mentioned. This evidence supports the technology adoption theories which form the theoretical basis for this study as summarized in Table 1.

Participants not overtly mentioning security as a requirement does not in any way detract from security being a critical requirement in eModeration. During the initial selection of constructs, security was recognized as a critical component. Despite no questions being structured around security, the lack of security in paper-based moderation was repeatedly discussed. Therefore, security is an essential requirement for an eModeration system.

Additional requirements which were not evident in the literature reviewed were phrased as follows by focus group participants:

* "Should allow for a voice over so that it is easy for the moderator to leave a comment."

* "Have a call button for both the assessor and the moderator to "meet" instead of back and forth communication i.e. multi-user communication technology."

# 4 Structured Requirements for an eModeration system

In determining the requirements for an efficient eModeration system, the broad categories of User Requirements (UR); Task Requirements (TR) and System Requirements (SR) were identified as depicted in Fig. 1 (see Rajamany, Van Staden and Van Biljon [26] for a full-sized diagram).

Based on the findings, it was deemed useful to replace *attitude* by *self-efficacy*. *Accuracy* was replaced by *audit trail* as this was the context in which participants viewed the accurate keeping of records. *Ease of learning* was removed as it was specific to literature. The assumption is that participants are skilled as well as experienced (cf. Table 2) in the use of technological resources. The survey results indicate that 70% of the participants believe that they have the capacity to learn to use a new system. The categorization of *flexibility* was based on the system supporting different file formats. It is thus necessary to support paperless work and other formats like videos [14].

Based on the context in which participants described constructs, *Productivity* and *Work* volume were merged into *Productivity*. *Job effects* and *Time consumed* were merged into *Time*. *Confidence* was removed and *Self-Efficacy* was retained and assumed to encompass confidence. After the analysis, core technological characteristics that were initially left out of the survey were added to the requirements. Context specific items, for instance a voice over button arising from the focus group interview, were added. The identified dimensions were then analyzed further. Dimensions present
in 2 or more streams of evidence (literature review, survey and focus group) were in-
cluded. These values were categorized under UR, SR or TR; with some overlap. For
instance, file formats was categorized as both SR and TR. User characteristics inform
UR. It was thus deemed necessary to include a separate dimension for user character-
istics.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Diagram depicting structured requirements</figcaption></figure>

5 Conclusion

This paper addresses the lack of eModeration requirements for secondary schools. A
single exploratory case study strategy was employed to determine the challenges, ben-
efits and requirements of an eModeration system with the purpose of identifying re-
quirements for an eModeration system. Empirical data was collected from a focus
group interview and an online survey to investigate the current moderation practices of
IT and CAT teachers at private secondary schools in SA and triangulate the findings
with the challenges, benefits and requirements abstracted from literature. The require-
ments for an eModeration system (presented under the categories of UR, TR and SR)
constitute the theoretical contribution of the study. The pre-adoption attributes could be
useful to designers and developers of eModeration systems and hence constitute a prac-
tical contribution.

6 Acknowledgement

This paper is based on the research supported by the South African Research Chairs
Initiative of the Department of Science and Technology and National Research Foun-
dation of South Africa (Grant No. 98564).
# 7 References

[1] G. Lynch, "GCSE ICT E-Moderation Pilot Case Study," the e-Assessment Association, 2014. [Online]. Available: http://www.e-assessment.com/category/resources/case-studies/page/3/. [Accessed: 25-Apr-2017].

[2] C. Van Staden, J. Van Biljon, and J. Kroeze, "Using a User Experience Evaluation Framework for eModeration," in 2017 Conference on Information Communications Technology and Society, 2017, pp. 1-6.

[3] New-Zealand-Qualifications-Authority, "Digital Assessment Requirements for Post-Moderation," ServiceIQ, 2016. [Online]. Available: http://www.serviceiq.org.nz/assets/Attachments/QA-Documents/SIQ-GUIDE-QA-PostModDigitalAssessment-Ed3-Aug16.pdf. [Accessed: 26-Apr-2018].

[4] A. Sorenson and L. Gill, "Digital Assessment Transformation: Internal Assessment and Moderation," 2015. [Online]. Available: http://www.nzqa.govt.nz/about-us/publications/newsletters-and-circulars/assessment-matters/digital-assessment-transformation-internal-assessment-and-moderation/. [Accessed: 21-Aug-2017].

[5] C. Van Staden, J. Van Biljon, and J. Kroeze, "eModeration: Towards a User Experience Evaluation Framework," in SAICSIT, 2015, no. September 28-30, pp. 1-11.

[6] C. Van Staden, "User experience evaluation of electronic moderation systems: A case study at a private higher education institution in South Africa," UNISA, 2017.

[7] New-Zealand-Qualifications-Authority, "Digital Moderation Discussion Paper," Wellington, 2016.

[8] AlphaPlus, "Standardisation methods, mark schemes, and their impact on marking reliability," 2014.

[9] C. Van Staden, "IT Moderation Going Green," in SAICSIT '10, 2010, pp. 426-428.

[10] M. Johnson and J. Greatorex, "Judging Text Presented on Screen: implications for validity," E-Learning, vol. 5, no. 1, pp. 40-50, 2008.

[11] L. Adie, "An investigation into online moderation," Assess. Matters, vol. 3, pp. 5-27, 2011.

[12] J. Greatorex, "Moderated e-portfolio project evaluation," Cambridge, 2004.

[13] S. Chamberlain and R. Taylor, "Online or face-to-face? An experimental study of examiner training," Br. J. Educ. Technol., vol. 42, no. 4, pp. 665-675, 2011.

[14] D. Berger, "Supporting Tool for Moderation in the Grading Process of Summative Assessments," Graz University of Technology, 2011.

[15] V. Venkatesh and H. Bala, "Technology Acceptance Model 3 and a Research Agenda on Interventions," Decis. Sci., vol. 39, no. 2, pp. 273-315, 2008.

[16] V. Venkatesh, M. G. Morris, G. B. Davis, and F. D. Davis, "User Acceptance of Information Technology: Toward a Unified View," MIS Q., vol. 27, no. 3, pp. 425-478, 2003.

[17] D. L. Goodhue and R. L. Thompson, "Task-Technology Fit and Individual Performance," MIS Q., vol. 19, no. 2, p. 213, 1995.

[18] T. Zhou, Y. Lu, and B. Wang, "Integrating TTF and UTAUT to explain mobile banking user adoption," Comput. Human Behav., vol. 26, pp. 760-767, 2010.

[19] B. Ramdani, D. Chevers, and D. Williams, "SMEs' adoption of enterprise applications A Technology-organization-environment model," J. Small Bus. Enterp. Dev., vol. 20,
no. 4, pp. 735–753, 2013.

[20] L. M. Erlirianto, A. H. N. Ali, and A. Herdiyanti, “The Implementation of the Human, Organization, and Technology–Fit (HOT – Fit) Framework to evaluate the Electronic Medical Record (EMR) System in a Hospital,” *Procedia Comput. Sci.*, vol. 72, pp. 580–587, 2015.

[21] B. Pynoo, P. Devolder, J. Tondeur, J. Van Braak, W. Duyck, and P. Duyck, “Predicting secondary school teachers’ acceptance and use of a digital learning environment: A cross-sectional study,” *Comput. Human Behav.*, vol. 27, no. 1, pp. 568–575, 2011.

[22] B. Rienties, B. Giesbers, S. Lygo-Baker, H. W. S. Ma, and R. Rees, “Why some teachers easily learn to use a new virtual learning environment: a technology acceptance perspective,” *Interact. Learn. Environ.*, vol. 24, no. 3, pp. 539–552, 2016.

[23] V. Venkatesh and F. D. Davis, “A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies,” *Manage. Sci.*, vol. 46, no. 2, pp. 186–204, 2000.

[24] D. L. Goodhue, “Development and Measurement Validity of a Task-Technology Fit Instrument for User Evaluations of Information Systems,” *Decis. Sci.*, vol. 29, no. I, pp. 105–138, 1998.

[25] M. M. Yusof, J. Kuljis, A. Papazafeiropoulou, and L. K. Stergioulas, “An evaluation framework for Health Information Systems: human, organization and technology-fit factors (HOT-fit),” *Int. J. Med. Inform.*, vol. 77, no. 6, pp. 386–398, 2008.

[26] V. Rajamany, C. Staden Van, and J. Van Biljon, “eModeration Requirements: A Case Study in Private Secondary Schools in South Africa,” 2020.

[27] C. Kivunja and A. B. Kuyini, “Understanding and Applying Research Paradigms in Educational Contexts,” *Int. J. High. Educ.*, vol. 6, no. 5, pp. 26–41, 2017.

[28] D. L. Morgan, “Pragmatism as a Paradigm for Social Research,” *Qual. Inq.*, vol. 20, no. 8, pp. 1045–1053, 2014.

[29] R. M. Dilshad and M. I. Latif, “Focus Group Interview as a Tool for Qualitative Research: An Analysis,” *Pakistan J. Soc. Sci.*, vol. 33, no. 1, pp. 191–198, 2013.

[30] R. A. Krueger and M. A. Casey, “Focus Group Interviewing,” in *Handbook of Practical Program Evaluation: Fourth Edition*, John Wiley and sons, 2015, pp. 506–534.

[31] V. Clarke and V. Braun, “Teaching thematic analysis: Overcoming challenges and developing strategies for effective learning,” *Psychologist*, vol. 26, no. 2, pp. 120–123, 2013.

[32] S. Fatimah, M. Yassin, N. S. Mohamad, and H. Yamat, “Developing W-Portfolio Culture in Computer,” in *Proceedings of the ePortfolio 2007 Conference. October, 17–19, 2007*, pp. 327–332.

[33] N. Raikes, J. Greatorex, and S. Shaw, “From Paper to Screen: some issues on the way,” *Int. Assoc. Educ. Assess. Conf.*, no. June, 2004.

[34] ABC-Awards, “eModeration Guide,” 2014. [Online]. Available: http://www.abcawards.co.uk/wp-content/uploads/2014/09/eModeration-Guide.pdf. [Accessed: 22-Mar-2018].