eExaminations development and acceptance

Andrew E. Fluck

► To cite this version:

Andrew E. Fluck. eExaminations development and acceptance. IFIP TC 3 International Conference on Key Competencies in the Knowledge Society (KCKS) / Held as Part of World Computer Congress (WCC), Sep 2010, Brisbane, Australia. pp.126-135, 10.1007/978-3-642-15378-5_12. hal-01054705

HAL Id: hal-01054705

https://inria.hal.science/hal-01054705

Submitted on 8 Aug 2014

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
eExaminations development and acceptance

Andrew E. Fluck

University of Tasmania
Locked Bag 1307, Launceston, Tasmania, 7250, AUSTRALIA
[Andrew.Fluck@utas.edu.au](mailto:Andrew.Fluck@utas.edu.au)

**Abstract:** Over three years students at the University of Tasmania have participated in eExaminations where they have responded to examination questions using computers. Over these three years they have made the transition from using institutional computers in laboratories to using personally owned laptops in a traditional examination hall. This innovation and transition have been paralleled and enabled by the development of a modified live operating system which preserves the fairness of the assessment process. The technical developments depended upon social innovation engineering efforts necessitated by the range of adopters amongst students and faculty. National recognition was achieved for this method which is being adopted more widely worldwide. The technical materials are available for download from [www.eExams.org](http://www.eExams.org).

**Keywords:** computer based examination, innovation engineering, Ubuntu, USB drive.

# 1 Introduction

With the explosion of eLearning over the last decade, there has been a corresponding surge of interest in computer mediated assessment [1]. Computers can facilitate a broader range of assessment techniques such as ePortfolios, blogs, discussion board responses and a range of digital recording techniques such as podcasting or video production.

These new techniques are themselves a smorgasbord of digital skills which add rich variety to a blend of assessment within any one course. Good assessment of student learning uses a combination of diagnostic, formative and summative evaluations. Summative assessment in particular needs to be valid, reliable and fair. The latter can be interpreted as meaning some part of the total assessment for a course should be done in such a way as to provide a reasonable level of assurance the submitted work is entirely that of the candidate who will receive an award. In some cases where students take home their coursework assessments, they are required to sign disclaimers such as "I hereby declare that I am the sole author of this paper and everything presented in this paper is my own" [2]. This does not enable the institution to be certain the work is in fact entirely due to the individual – some may
have friendship and professional networks which help them develop ideas more than other candidates.

For assessments where self-certification is inappropriate, more formal examinations are conducted. As digital computers become more widely available to pupils in schools [3] and students in universities, quality assurance organizations are looking at electronic assessment methods more closely. In Norway *The Knowledge Promotion* reforms have led to the acceptance of protocols of computer-based examinations in high schools and senior secondary colleges [4].

Leister, Fretland and Solheim [5] described how the 19 counties of Norway have adopted strategies to incorporate computer based examinations. A common assumption is that candidate identity will be checked and activity supervised during the examination by non-technical staff, often retired teachers.

When considering computer-based examinations there are a range of technological implementation aspects. These include ways in which questions should be delivered and answers collected (reticulation). The technology chosen will reflect a policy balance between security, convenience and reliability. This kind of policy response and those to other problems associated with computer-based examinations are described in Table 1.

<table><caption>**Table 1.** Areas of policy concern for computer-based examinations in schools</caption><thead><tr><th>Area of concern</th><th colspan="3">Opposing policy stances</th></tr></thead><tbody><tr><td>Reticulation of questions and answer responses</td><td>Networked</td><td>↔</td><td>Standalone</td></tr><tr><td>Computer ownership</td><td>Institutional</td><td>↔</td><td>Personal</td></tr><tr><td>Computer functionality</td><td>Kiosk (locked) mode</td><td>↔</td><td>Wide range of software (sometimes using virtualization or a compatibility layer)</td></tr><tr><td>Candidate communication</td><td>Function blocking</td><td>↔</td><td>Logging/monitoring</td></tr><tr><td>Candidate familiarity</td><td>Common learning environment</td><td>↔</td><td>Test environment requires familiarization/training</td></tr><tr><td>Licencing costs</td><td>Commercial</td><td>↔</td><td>Free, open source software</td></tr></tbody></table>

Each of the issues in Table 1 is now addressed in more detail.

The whole area of question and answer reticulation requires a fault tolerant approach to reliability of the examination system. Operational communication facilities during the examination allow answers to be progressively backed up, providing a measure of redundancy if a single candidate's equipment should fail. However, any operational communication link could be exploited by a mischievous candidate for the purposes of cheating, and this creates a potential security loophole. Furthermore, equipment failure at the level of a school wireless access point or router,
or at the systemic server centre, or any point in between, could interrupt examinations for a very large number of candidates.

As more students and pupils acquire their own computers (laptops or netbooks) and are permitted to connect them to school networks, there is less pressure to provide information technology equipment within the institution. However, these machines may be prepared with special software to interdict examination protocols. Their use may be the only economic way of overcoming the logistical problems of supplying a large number of workstations required for examination purposes.

Generally, running an examination within a web-browser window (kiosk mode) requires the questions to be programmed using *Shockwave Flash* or a similar programming language. This may be very suitable for providing question tracking to each student during the examination, but is only viable for very large scale assessments with a long time-frame for preparation. Giving candidates access to the full operating system can allow the use of specialist software within the assessment process.

For the examination to be fair, it is generally accepted that candidates should not be able to communicate with one another or any other person. Additionally, digital communication beyond the location of the exam could be used to access pre-prepared answers or other helpful information which may not be equally accessible to all candidates. Function blocking works on the candidate's equipment or in a local router to make sure only permitted communications occur such as with a question server or answer backup database. The alternative approach is to allow all communications, but to log them and alert local supervisors if there is an exceptional message.

Candidate familiarity with the selected system appears to be a common request from awarding bodies, but is rarely well defined. In some case this has been interpreted as meaning Microsoft Windows (without actually specifying a commercial product) since many pupils use this operating system for their learning. There is certainly a good case for students being able to undertake standard operations such as launching software, entering text etc. without cognitive hurdles.

Whichever system is used by candidates for their exams, it must be within legal guidelines. If an operating system requires a licence fee for each installation, then this fee must to be paid. Such an economic argument is a strong one for basing a system on whatever software is already installed upon the machine each candidate uses.

In the Norwegian county of *Møre og Romsdal* the system adopted for trials was a networked, personal, wide range with function blocking in a novel test environment using FOSS (free, open source software). It has been argued that if learners conventionally use Microsoft Windows and Office, then the latter two areas of concern are in tension. It could equally well be argued that a FOSS environment for which the candidate has received familiarisation training can eliminate this tension.

## 2 Method

Our objective in this project was to determine the viability of a new approach to computer-based examinations in tertiary education. We commissioned the construction of a special version of the Ubuntu operating system and application
software for the purpose. This eExam System was run on target computers by
inserting a CD-ROM (or later, a USB drive) and holding down the ‘one-time boot
menu’ key on power-up. This action generates a menu of bootable sources and the
candidate selects the eExam disk or USB drive.

The investigation is described by a participant observer using a stepwise
improvement method. In the context of rapidly evolving technology, any constant
comparison approach would rapidly become unviable. In this sense the eExamination
system has been updated by our support staff at Open Technology solutions<fnref n="1" /> in
Hobart as new versions of Ubuntu have been released, retaining core functionality at
each stage.

The initial trials of the eExamination system have used only some of the potential
that moving to computer-based assessment entails. Using computers can support
complex, authentic, real-life contexts and activities in which candidates can
demonstrate multiple learning perspectives. However, there are few examples of such
digital assessment practices [6].

Speculation about intelligent computer-aided assessment [7] has not been
considered in this research. This area is one which shows much promise, but would
require an autonomous agent running between the operating system and candidate-
initiated applications.

Additionally the method can be viewed as action research presented as a case
study.

## 2.1 The three trials

### 2.1.1 First trial

Trials began in 2007 at the University of Tasmania using a modified version of Ubuntu 7.04 and university lab-based computers. The unit of study was in the third year of a Bachelor of Education degree, and was intended to equip pre-service teachers to use ICT in their classroom practice. As with all the following trials, every student was issued with a CD-ROM and an example eExamination very early in the unit of study. The IT support staff agreed to waive normal operating conditions to allow one laboratory of computers to boot from this external operating system source, and students were shown how to activate the boot-menu of any personal computer at home. In this case Open Office and other software (such as The GIMP for creating diagrams) were on the CD-ROM together with the question file.

Our modified version of Ubuntu did not contain the IP stack, preventing internet connectivity using any kind of networking. To ensure the system was correctly loaded, a unique photograph was displayed on the desktop background of each candidate's computer. This could be quickly checked by the supervisors without technical knowledge. The software also prevented any access to the local hard disk

<footnote n="1">¹ http://www.opentechnologysolutions.com.au</footnote>
drive, so digitally based system modification or access to previously prepared answers
was prevented.

During the eExamination, students accessed videos and other materials for each question from the CD-ROM, but saved their answer files to the computer desktop. At the end of the assessment, a USB stick was passed from student to student to capture the finished answer files. The files were burned onto a CD-ROM before passing to various markers. Post-exam surveys showed students were divided between those that liked the new ICT-based format with 56% preferring to have formal tests conducted using computers instead of using handwriting on paper.

## 2.1.2 Second Trial

In the 2008 trial the eExamination was modified to be presented in two files. The first was a PDF document (therefore difficult for candidates to modify) and the second a mostly blank text document file with spaces for each answer response. Post-exam focus groups showed that the multiplicity of windows (some questions involved video or other files of pupil work) was very confusing. The process was altered slightly to allow students with their own laptops to bring those into the eExamination and use them in the same room as candidates using institutional desktop computers [8].

Only 35% of students in the post-exam survey preferred computers, and this was attributed to the greater complexity of handling multiple windows. A focus group identified dichotomous responses amongst the candidate cohort. Touch-typists considered themselves advantaged in the new situation, whilst those whose handwriting was considered faster, felt discriminated against.

## 2.1.3 Third Trial

An important change was made in the third trial by gaining agreement from the University Exams Office that the eExamination would take place in the same hall as all the other exams. There were two conditions – that the eExamination would be scheduled after all the other exams, and would be the only exam at the time. An inspection showed the hall had 21 mains electricity power points. Surge suppression devices were provided with outlets for up to six laptop mains adapters on each power point. Most students would use their own laptop computers, with the institution providing loan machines for the residue (about 10% of the cohort).

# 3 Results

These results relate to the third trial, where most students provided their own personal computer. When learning about the eExamination near the beginning of the teaching unit, students accepted the idea. As the time for the assessment drew closer however, some started to express apprehension. One of the lectures focussed on theories of innovation adoption, and they could see various levels of preparedness within the group, ranging from early adopters to laggards.
Students used the anonymous discussion board in the learning management system to sound out their worries:

Having talked to a number of people, it's obvious a lot of us have concerns about using our own laptops in the exam. It would be excellent if everyone could let their opinions be known here... I think it puts some of us at a disadvantage. To begin with not everyone HAS a laptop, what about them? Also, batteries and charging are an issue depending on the type of laptop being used. Not to mention that some computers take longer to load than others...

These difficulties had already been solved by the arrangements above. However, the laggard section [9] of the cohort was quite strident in identifying objections:

I have heard that there are differences in the running speed between the CD version and the USB version. I don't recall ever being in an exam where the type of booklet given determines the speed you can complete the exam. I think the real issue is that the Faculty wants to be the first to have an exam using laptops.

The students were experiencing a halo effect, and were becoming aware that this was a novel practice. This added to their concerns, and forced them to enquire deeply into the new system. Since it had not (to our knowledge) been used before, it was impossible to prove all their concerns would be addressed. Also, what were the metrics for comparison? This tension was expressed in this exchange between an anonymous student and the unit coordinator:

> I am totally for using a laptop for the exam, absolutely. But only if it can be proved that the conditions will be as equitable as those during hand writing exams.

The speed question is an interesting one. You can look at it two ways. Firstly, the time required to boot up and get going. This can vary according to how fast your computer is (whether you are using USB or CD for instance), but won't affect the time you have because no-one will be allowed to start writing until everyone is booted up. The second is the speed you can enter your answers. Most of this is typing, and I doubt ANYONE can type faster than the slowest computer can accept text. So once again, you are on a level playing field.

This student uncertainty was communicated to the university administration, and several questions were asked by the head of school about the process. Fortunately the Faculty was supportive, and continued to permit the eExamination to proceed. Some students wrote strongly in favour of the course and the exam mode:

> I also do not feel that our concerns were going to change requiring a laptop to sit the exam, given much of Dr Fluck's research involves investigating electronic testing and examination. At the same time, I also feel that if certain people had accepted this earlier, then the valuable lecture and tutorial time could have been used discussing the content instead of answering tedious laptop preparation questions that
had already been asked over and over. Complaining about and
discussing this seemingly non-negotiable point has added to the
perceived stressful occasion of sitting an exam, having inspired panic,
fear and anxiety. Generally this has affected people whose ICT skills
are lagging, those who need the most confidence leading up to the
exam. I hope they do not regret not expending their energies in a more
constructive way. I trust everyone involved, both staff and students, will
look back on this experience subjectively as a first hand understanding
of some of the realistic difficulties involved in implementing IT. In this
life, sometimes you just have to do the best with the limitations placed
upon you. Best of luck everyone.

The examination materials were presented in a new fashion to reflect the concerns of the second trial cohort. A single document file was provided which contained both the questions and spaces for answers. The questions were converted into images and the file placed in a read-only section of the USB stick issued to every candidate. This had two effects. Firstly, students could not type in answers until they had re-saved the file in the answers partition of the USB stick. Secondly, although a question might be inadvertently deleted, it would disappear in its entirety – thus preventing the subtle one-word alteration which might detract from the problem! If such an event took place inadvertently, the original file remained for reference.

The eExamination took place with only minor hitches. All students were provided with a USB stick containing the bootable modified version of Ubuntu, the questions and security desktop picture, and a blank partition for answers. Some had previously arranged to get CD-ROM versions of the bootable operating system. A few student laptops had USB (universal serial bus) sockets which were so indented the eExamination stick would not enter. These two students completed the exam using paper backups. One student had a complete computer malfunction close to the end and was required to re-sit the examination. This event has forced a re-think of the procedure for recording answers to ensure a backup remains on the USB stick. Student answer scripts were copied from the USB sticks and were then burned to CD before distribution to markers. Markers chose to assess on screen, annotating the scripts with marks awarded, and collated totals into a spreadsheet.

A week after the eExamination had been successfully completed, a national
newspaper flew a photographer to Tasmania and a story was printed on page two of
the higher education supplement [10]. It described how 124 students on three
University of Tasmania campuses took their laptops into their exam rooms to sit what
was thought to be the first tertiary eExam in Australia. It not only changed the way we
assessed students, but also paves the way for changing what we teach.
<figure><img src="image_2.png" /><figcaption>**Fig. 1.** A set of USB sticks each containing the eExamination system, question paper and answer partition</figcaption></figure>

# 4 Discussion and conclusion

This use of a live operating system for providing a fair environment for examinations is not unique. The DigEks software is based on Kubuntu, and works in a similar way. It is more flexible than the current eExam System, in that both stand alone and restricted networked environments are catered for [5].

What are the lessons that can be learned from this case study?

Firstly, the impact of ICT on learning, even at the tertiary level, is not well understood by participants. In this case the participants were a 'cusp' generation, moving from the classroom practices they experienced as children, to creating new practices in their future roles as teachers. They are therefore a vital ingredient in transforming learning to a digital curriculum.

The first two trials and their impacts have been discussed in fuller length [8]. The move to personal laptops was seen by some candidates as the final straw – this was an exam (unusual in their course); it was computer-based (a unique situation); and furthermore they were expected to provide the computer themselves (costly and unheard of throughout the country). These appeared to be the main reasons for their apprehension.

The high noise levels reported by students in the first trial were quite absent from the third trial. Whereas institutional desktops in confined laboratories echoed to the noisy keyboards, the softer laptop keyboards in a large gymnasium space in the third trial were as quiet as pens on paper.

A critical question is the use of stand-alone computers compared to networked equipment. In the Norwegian case study [11] a systemic requirement to have
restricted or monitored internet access required networking to be retained for the
reticulation of questions and backup of answers.

Arguments for keeping active networking during the examination focus upon the
ease of distributing questions and collecting answers. However, this comes with a
penalty. Firstly, as discussed in the introduction, this ease of use increases the risk of
unreliability because many infrastructure components must operate flawlessly
throughout the assessment. A single point failure can affect many, if not all, the
candidates. Also, active networking can also be exploited by cheats, or cause
inadvertent exception alerts through the operation of software (such as operating
system updates) without action by the candidate.

The trials at the University of Tasmania have used stand-alone computers for eExaminations with no networking at all. This has eliminated many causes of unreliability but has transferred the reticulation system to USB sticks. Hitherto a single person has formatted each stick in preparation, and transferred the answer files to CD-ROM before marking. Future trials will use a USB duplicator with a capacity to download all answer files into a controlling computer. With large batches of 20 sticks dealt with in minutes using this system, local reticulation becomes tolerant of time delays or intermittent electricity supply. On a larger scale the encrypted distribution of questions to the local venue some time before the eExamination could be mixed with the local programming of USB sticks.

Because the operating system is run from the examination CD or USB stick, this
system is agnostic with respect to the ownership of the computing equipment.
Institutional computers can be used just as easily as candidate-owned laptops.

The functionality of a kiosk-based system is highly restricted, and therefore full
access to the operating system has greater potential to foster a digital curriculum. For
example, if candidates were asked to prepare an analysis of economic trends using a
spreadsheet, this would be difficult to do within a web-browser window without
sophisticated software support. Ubuntu releases contain Open Office, which contains
this kind of general purpose tool, together with the GIMP, a package for creating and
manipulating pictures. If a more specialist piece of software (for computer-aided
design for instance, or modelling chemical reactions) this could be included in the
eExamination system. Linux-based software would be easy to include, and Windows-
based software can often run under WINE<fnref n="2" />.

As discussed above, the eExaminations system omits all computer-based
communication, and therefore peer to peer messages are not possible, preventing
collusion.

Some jurisdictions require candidates to have access to a 'familiar environment' when using computers for examinations. It could be argued that habitual users of Apple Macintosh computers would be disadvantaged if required to use a Microsoft Windows PC just for their exam – and vice versa. The process adopted at the University of Tasmania has been to briefly demonstrate the Ubuntu system to prospective candidates, and make a copy available for familiarisation over a month before the eExamination. This would not be permitted under the commercial licensing arrangements of the other two systems. Students have the chance to try it on their

<footnote n="2">² WINE is a translation layer (a program loader) capable of running Windows applications on Linux or other operating systems.</footnote>
own computer (perhaps the one they will use for the exam) or on an institutional
computer to undertake a previous year's questions.

Another advantage of the open source nature of the eExamination system based on Ubuntu, is that teaching staff can also be given copies for preparation of new examinations. These advantages have been embraced for further trials in the Faculties of Law, Arts (for History examinations), Science (for Architecture responses to detailed photographs of buildings) and potentially Human Life Science where movies of items in the pathology museum can be seen on screen with magnification.

These advantages for eExaminations can help make current exams better. However, in the longer term, this new medium for assessment holds promise which is far more significant. When the culminating summative assessment is conducted using a computer, candidates can be required to use sophisticated software analogous to that used by a practicing professional. For instance, engineers may be required to assess structures using a complex set of integral calculus equations. Software such as Maxima, Maple or Mathematica could be made available during the assessment to make such a complex problem soluble. In changing the way assessment is conducted, teachers can choose to alter course content to deal with topics in more authentic ways or at significantly higher levels of complexity. This is what is meant by the emergence of a digital curriculum.

In conclusion, the emergence of a digital curriculum appears to depend upon the availability of a digital assessment environment which is flexible, fair and scalable. Several systems can be expected to emerge in school education and tertiary learning institutions. The nature of the digital curriculum, its core content and the breadth of learning it embraces will all depend upon the characteristics of the eExamination system that becomes prevalent.

## References

1. Backroad Connections Pty Ltd: Assessment and Online Teaching (Version 1.01): Australian Flexible Learning Framework Quick Guides series. Australian National Training Authority. http://pre2005.flexiblelearning.net.au/guides/assessment.pdf (2004)

2. Moos, D.: EDHD 420 Cognitive Development and Learning. University of Maryland. http://www.education.umd.edu/EDHD/course/syllabi/fall06/edhd420.pdf (2006)

3. Trucano, M.: Uruguay's Plan Ceibal: The world's most ambitious roll-out of educational technologies? EduTech, World Bank. http://blogs.worldbank.org/edutech/videos/uruguays-plan-ceibal-the-worlds-most-ambitious-roll-out-of-educational-technologies 30 Jan 2010. (2009)

4. Norwegian Directorate of Education and Training: Hjelpemidler til eksamen I Kunnskapsløftet. http://www.udir.no/upload/Eksamen/Hjelpemidler_til_eksamen_i_kunnskapsloftet.pdf, in Norwegian, September 2, 2009, last accessed January 30, 2010 (2008)

5. Leister, W., Fretland, T.: How to Perform ICT-based Exams in Compulsory Schools. Norwegian Computing Center. Presentation at NUUG mote, 10th November 2009, Oslo / Stavanger. (2009)

6. Cook, D.: Views of learning, assessment and the potential place of IT. In Anne McDougall, John Murnane, Anthony Jones and Nick Reynolds (eds) Researching IT in
Education: Theory, practice and future directions. Routledge, London. (2010)

7. Bescherer, C., Spannagel, C., Kortenkamp, U., Muller, W.: Research in the field of intelligent computer-aided assessment. In Anne McDougall, John Murnane, Anthony Jones and Nick Reynolds (eds) Researching IT in Education: Theory, practice and future directions. Routledge, London. (2010)

8. Fluck, A., Pullen, D., Harper, C.: Case study of a computer based examination system. Australasian Journal of Educational Technology 25(4)509-523. http://www.ascilite.org.au/ajet/ajet25/fluck.html (2009)

9. Rogers, E.M.: *Diffusion of innovations* (5th ed.). Simon and Schuster, New York. (2003)

10. Lane, B.: Laptops pass the big exam. The Australian (newspaper). Sydney, 18 November p.24. (2009)

11. Leister, W., Fretland, T., Solheim, I.: Preventing unwanted communication in ICT-based exams by using free software. Norwegian Computing Centre. http://publications.nr.no/nokobit2009_finalise_2.pdf (2009)