The Electronic Journal of Mathematics and Technology, Volume 8, Number 5, ISSN 1933-2823

Item Bank System for the Mathematics e-Learning
System STACK
Yasuyuki Nakamura
nakamura@nagoya-u.jp
Graduate School of Information Science
Nagoya University
Japan

Tetsuya Taniguchi
tetsuya@kitasato-u.ac.jp
College of Liberal Arts and Sciences
Kitasato University
Japan

Takahiro Nakahara
nakahara@3strings.co.jp
Sangensha LLC
Japan
Abstract: System for Teaching and Assessment using a Computer Algebra Kernel (STACK) is a computer-aided
assessment system for mathematics, which functions as a plug-in for the learning management system Moodle. Using
STACK, Moodle can provide questions for online mathematics tests. These questions typically require a mathematicalexpression as a response and the responses are assessed algebraically. STACK can be used to determine whether astudent’s answer is correct, as well as providing appropriate feedback for various answers. Partial scores can even be
provided for an incomplete answer. However, the questions must be designed carefully to take full advantage of
STACK’s rich functionality. Well-designed questions can be valuable educational resources, especially in an e-learning
environment. Thus, to promote the sharing of high-quality questions for use in online tests, we developed an Item Bank
System for the mathematics e-learning system STACK. Using our system, users can submit authored questions in XML
format to STACK along with additional metadata, including the subject, difficulty level, targeted grade and publicity
level. After they have been uploaded and stored in the system, all users can utilize any of the questions that are open to
the public. We expect that sharing these questions will further promote the effective use of mathematics e-learning
systems.

1. Introduction
In recent years, information and communication technology (ICT) infrastructures have
improved in schools. Many now offer computer suites to students so they can gain practical work
experience and Wi-Fi networks are available to allow students to access the Internet using their own
laptop computers. Portable digital devices, such as smartphones and tablet computers, are also
gaining popularity and they are now ubiquitous in educational environments. The improved ICT
infrastructure and the presence of these devices mean that e-learning has become increasingly
popular. One of the most important functions of an e-learning system is the ability to administer
online tests to evaluate a student’s comprehension of the course material. Typically, students readthe test questions and enter their answers using an online form, where the answers are assessed
automatically by an e-learning system. However, most existing online tests are only capable ofassessing answers submitted in a true/false, multiple choice, numeric value, or simple descriptive
format. Only a small minority of systems can assess the validity of algebraic mathematicalexpressions. Maple T.A. and System for Teaching and Assessment using a Computer Algebra

The Electronic Journal of Mathematics and Technology, Volume 8, Number 5, ISSN 1933-2823
Kernel (STACK) [1, 2, 3, 4] are two examples of such systems. Maple T.A. is a commercial
software product, which can be accessed from a wide range of learning management systems(LMS) such as Blackboard and Moodle. STACK is integrated with Moodle and is open source.
We have been using Moodle as a LMS in our class and we employ STACK to administer
online tests that involve mathematical expressions for science and maths subjects. As we show
later, STACK has a rich set of functions that support science education. For example, it is capableof providing appropriate feedback in response to a variety of answers. However, there are several
problems associated with utilizing STACK in a classroom setting. One of these is the complexity ofauthoring test questions, where it is more difficult to create higher quality questions. In order toovercome this issue, we developed a question authoring tool for STACK [5]. We consider that welldesigned questions are a valuable teaching resource and sharing them will reduce the cumulative
effort required to author high quality questions. To facilitate this, we decided to develop an ItemBank System for use as a platform to share questions or other items.
The remainder of this paper is organized as follows. In the next section, we provide a brief
review of STACK and several examples of questions. In Section 3, we discuss several problems
associated with STACK and explain how to resolve these problems. We then introduce the ItemBank System and describe how it operates in Section 4. In the final section, we provide ourconclusions.

2. Brief review of STACKSTACK is a computer-aided assessment package for mathematics, which was developed by
Sangwin at the University of Birmingham, UK. STACK v.2 is fully integrated with Moodle, aLMS. The integration was achieved in SOAP and it gives users the option of including questions in
online tests that treat mathematical expressions as answers. The latest version of STACK is
compatible with Moodle’s quiz module and it is much simpler to install than previous versions.
When we evaluate a student’s understanding of a scientific subject using an online test,
simple questions types such as multiple choice questions (MCQs), which are often included in
online tests, are not sufficient because these questions formats cannot accurately represent a
student’s comprehension level. For example, when answering MCQs, students can simply “choose”
the correct answer from a list. Even if the student does not know the correct answer, there is apossibility of choosing the correct answer by guessing. Thus, it is possible to evaluate a student’s
overall comprehension better when a large number of questions are presented to a student.
However, it is unclear whether the student understood each question correctly within the context ofthe test. Therefore, questions that require students to input manual responses in the form ofalgebraic expressions are more accurate for evaluating comprehension, especially in scientificsubjects. In order for this approach to be implemented, the mathematical expressions provided by
students would have to be assessed algebraically. However, only a few systems are capable of thisand STACK is one of them. The need for and applications of computer-aided assessments inscientific subjects have been discussed in detail previously [3].
Using STACK, students can provide answers to test questions in the form of mathematicalexpressions (e.g. polynomial expressions, matrix expressions, and functions). STACK evaluates theanswers provided and generates responses that confirm or deny the accuracy of the answer, as wellas giving valuable feedback. STACK uses Maxima, a computer algebra system, to assess themathematical expressions provided by students.
Figure 2.1 shows an example of a simple differential equation, i.e.,dy+ 2y = 0 ,
dx
356

(2.1)

The Electronic Journal of Mathematics and Technology, Volume 8, Number 5, ISSN 1933-2823

which was given to students as an online test in our class. The question is quite simple, but it
includes two elements that need to be reviewed in order to evaluate the students’ understanding, i.e.,
it is necessary to check for the presence of any miscalculations and to determine whether anarbitrary constant was added. The correct answer to the question is Ce-2 x , which would be enteredas C*exp(-2*x), but there are two typical incorrect answers, i.e., C*exp(2*x), which is an
example of a miscalculation, and exp(-2*x), which does not include an arbitrary constant. Thus,
we can frame statements that provide feedback. Examples of this are “Incorrect answer. Youranswer should satisfy the ODE, but it does not.” if the answer was miscalculated, and: “Youranswer is partially correct. The solution should contain a constant but your answer does not.” if theanswer did not contain a constant (see Figure 2.2).

Figure 2.1 An example of a question that asks students to solve a simple differential equation.

Figure 2.2 Examples of feedback provided for incorrect (left) and partially correct (right) answers.
In the first example shown in Figure 2.2 (left), the students can ascertain why their solutions
are incorrect by reading the feedback provided by the system. They can learn that a solution to adifferential equation should satisfy the differential equation. This is a very basic concept indifferential equations, but some students do not recognize it immediately. A solution to adifferential equation should contain arbitrary constants in response to the order of the differential
equation as a general solution. However, students often forget to include constants. Therefore, thefeedback shown in the second example of Figure 2.2 (right) can draw attention to this common
mistake. Suitable feedback can be derived by comparing answers with a “potential response tree”,
which can provide feedback to students based on their responses. An example of a potential
response tree is illustrated in Figure 2.3. This functionality is one of the most distinctive features of
STACK. In order for the system to operate, teachers must take the time to determine potential
responses, which is a difficult, but necessary, task. If a question has higher complexity, it is moredifficult to create the potential response tree.

357

The Electronic Journal of Mathematics and Technology, Volume 8, Number 5, ISSN 1933-2823

Yes

No

Yes

No

Figure 2.3 Potential response tree for the question in Figure 2.1.
Let us consider other examples. Figure 2.4 shows an example of a complex potential
d2ydy
response tree for a second order differential equation+ 4 - 5y = 0 , where the solution is
2
dx
dx
x
-5x
y(x) = Ae + Be . Many potential answers might be provided by the students, which are all listed
in the potential response tree. Thus, the tree should be well designed or well structured to give
suitable feedback and reasonable scores, which helps the students to study by themselves using themaths e-learning system STACK. Well-designed questions can be valuable educational resources,
especially in an e-learning environment. Figure 2.5 shows an example of a question in the subject
area of mechanics. There are three answer fields for this question. In order to answer the subquestions, the students should understand the law of conservation of momentum and the law ofconservation of energy. Therefore, the use of well-designed questions that require more than one
concept, as well as the inclusion of a mechanism to give suitable feedback, allow student to learn
each concept and the relationships among these concepts.
When assignments are given to students, their work must be graded and returned to them aspart of the education process. A computer-aided assessment system such as STACK has thepotential to help teachers save time because of its feedback system and grading functionality.
Questions that have been designed carefully to provide feedback, such as the one shown in Figure
2.2, can yield significant educational benefits. Therefore, sharing questions with educational value
is important in an e-learning environment.
We have provided a brief summary of STACK. It is important to note that STACK iscurrently used by many institutes all over the world [2], which indicates its usefulness.

358

The Electronic Journal of Mathematics and Technology, Volume 8, Number 5, ISSN 1933-2823

YesNo

Figure 2.4 An example of the potential response tree for a second order differential equation.

Figure 2.5 An example of a question from the subject area of mechanics.

3. Problems and solutions using STACKAs described above, STACK can provide a rich set of functionalities to aid scienceeducation. However, the system is not without its problems. In this section, we review some of themain issues associated with STACK and describe how we solved them.
The first problem is related to the installation of STACK on a computer. STACK comprises
two main external tools: Maxima, a computer algebra system that is used to assess students’answers algebraically; and gnuplot, a function-plotting tool that is used to draw graphs. Maxima
requires Common Lisp and the problems that usually arise are related to the compatibility of thesetools. We solved this problem by writing a document that describes the best combination of tools
and by preparing binaries for the tools on our server [4]. The second problem with STACK is thelanguage environment it uses. STACK’s language environment is set independently from the oneused in Moodle. This situation is inconvenient because STACK is a subsystem of Moodle. We
made minor changes to STACK’s source code when we localized it for Japanese [4]. Following our
359

The Electronic Journal of Mathematics and Technology, Volume 8, Number 5, ISSN 1933-2823
modifications, STACK’s language environment is inherited from Moodle instead of being
independent. We are preparing a Japanese language pack for the latest version of STACK. The third
issue with STACK was its processing speed. From a user’s perspective, it can take a long time forthe system to process answers, thereby affecting the attention of students. This problem washandled in the version 2.2 release of STACK, where the processing speed was improved
significantly using caching. The fourth issue is that there is no perfect input method formathematical expressions. This issue is yet to be solved, but it should be addressed in the future.
The final major problem associated with STACK is the topic of this paper. As we mentioned
above, STACK can provide appropriate feedback to specific types of answers, which is one of its
most important functions. However, authoring questions is elaborate and time-consuming work,
especially when we create questions with more complex potential response trees, as shown in
Figure. 2.4. Nevertheless, the current interface for authoring questions is not convenient to use. In
response, we developed a question-authoring tool using Microsoft Excel (Excel) and Visual Basic
for Applications (VBA) [5]. In order for questions to be effective at promoting education, they mustbe formulated well. If well-structured questions are shared, thereby allowing any registered user
access to them, this would solve the fifth problem in an effective manner, and this could help
broaden the applications of e-learning to maths and science subjects. In order to make a platform forsharing questions, we developed an Item Bank System.

4. Development of the Item Bank System
There are three key phases in the development of an Item Bank System: the collection phase(collecting questions), use phase (questions are used) and build-up phase (the Item Bank Systemevolves). In this study, we outline the development of Mathbank.jp, http://mathbank.jp/en/, which is
our Item Bank System.
4.1 Collection Phase
In order to share questions, it is necessary to collect them first. Thus, it is essential to createa simple, user-friendly interface to encourage users to register questions. We developed the ItemBank System shown in Figure 4.1 as a Moodle system. Any user can register questions using the“Regist” link and search for registered questions from the “Search” link after user authentication.

Figure 4.1 Front page of Mathbank.jp.
360

The Electronic Journal of Mathematics and Technology, Volume 8, Number 5, ISSN 1933-2823
After selecting “Regist”, the registration screen shown in Figure 4.2 opens. When users
register a question in the Item Bank, they should include metadata such as the grade, difficultylevel, publicity level and key words. Questions can be registered by uploading an XML file via theinterface. Alternatively, existing STACK questions on the Moodle server can be registered to theItem Bank System by adding metadata information. This makes it possible to register and classify
existing questions in an effective manner. Metadata elements can be set from an administration
screen. Both STACK question types and general Moodle quizzes as a gift format can utilize theItem Bank System. Users can search through the questions in the system by selecting the “Search”
link and perform searches based on the metadata specified. A user can create questions on the ItemBank System itself by selecting the “Create” link and various tools, e.g., a question-authoring tool
developed using Excel and VBA [5] is provided on the site after selecting the “Tool” link.

Figure 4.2 Screen for registering questions.
4.2 Use Phase
The use phase provides time for the Item Bank System to be tried and tested. Aftersearching through the list of questions, users can download STACK questions in an XML format.
Users can then import the file to their servers for later use. The use phase also provides time forregistered questions to be tested straight from the Item Bank System, which creates stored logs. The
log is used to reconsider the difficulty level and to improve the quality of the questions in thesystem.
4.3 Build-up Phase
The build-up phase aims to improve the Item Bank System further based on stored data, as
well as adding value to collected questions that have not yet been implemented. For example, we
can review the responses to specific questions to generate a list of incorrect responses to include ina MCQ. This type of question may be more suitable for mobile devices. Furthermore, this phase361

The Electronic Journal of Mathematics and Technology, Volume 8, Number 5, ISSN 1933-2823
allows time to gather trial data, which could determine the true difficulty level of questions moreaccurately, because this is set objectively when each question is first registered.
4.4 Feedbacks from Users
About six months have passed since we opened the Item Bank System to the public.
Nineteen users and about 40 questions were available on the system as of April 13, 2014. Many
users have said that it is convenient to share questions that are listed on the system because they can
produce other questions based on the shared questions, which results in time savings. However,
metadata such as the grade, difficulty level, publicity level and the key words used to classify theregistered questions should be considered based on the feedbacks from users. For example, if weclassify questions based on the grade metadata, the grade is not common to all schools because theprogress schedule depends on each specific school. Some users have said that the questions should
be classified based on the subject.

5. Conclusion
One of the most distinctive features of STACK is the ability to provide pertinent feedback tostudents taking online tests by analysing their responses using a potential response tree. Authoringquestions that are suitable for the STACK framework may be difficult and time consuming, butthey are also valuable educational resources. We consider that the cumulative effort required tocreate suitable questions might be reduced if quality questions were shared freely among educators.
In order to facilitate this, we developed the Item Bank System, which is a platform for sharing
questions. This means that anyone can register questions or download questions for their own use.
The system was designed to be compatible with existing Moodle questions.
In order for this system to work properly, we made several modifications to STACK,
including enhancing its plotting features [6]. The Item Bank System is a major development for
STACK because creating high-quality questions is one of the most important tasks during elearning using STACK. Furthermore, some users have considered sharing questions with othermathematics online testing systems as well as STACK, e.g., MATH ON WEB Learning College
Mathematics by webMathematica [7], which should be promising future work. We expect thatsharing questions will further enhance and broaden the scope of e-learning in the future.
Acknowledgement This research was supported partially by KAKENHI Grant-in-Aid for Scientific
Research (C) 25350353 from the Japan Society for the Promotion of Science (JSPS).
References[1] Sangwin, C. J. (2004). Assessing mathematics automatically using computer algebra and theInternet. Teaching Mathematics and its Applications, 23(1), 1-14.
[2] Sangwin, C. J. (2010). Who uses STACK? A report on the use of the STACK CAA system,
Retrieved Sep. 9, 2013, from http://mathstore.ac.uk/headocs/WhoUsesSTACK.pdf.
[3] Sangwin, C. (2013). Computer Aided Assessment of Mathematics. Oxford University Press.
[4] Nakamura, Y. (2010). Mathematics e-Learning (in Japanese). Tokyo, Japan: Tokyo Denki
University Press.
[5] Nakamura, Y., Ohmata, Y. & Nakahara, T. (2012). Development of a question-authoring tool formath e-learning system STACK. IADIS International Conference e-Learning, 2012, Lisbon,
Portugal, 435-440.
[6] Nakamura, Y., Amano, H. & Nakahara T. (2011). Enhancement of plotting function of math elearning system STACK. World Conference on E-Learning in Corporate, Government, Healthcare,& Higher Education, Honolulu, U.S.A., 2621-2627.
[7] MATH ON WEB Learning College Mathematics by webMathematica, http://www.las.osakafuu.ac.jp/lecture/math/MathOnWeb/362