# A Conceptual Framework for Automatic Generation of Examinations Using Machine Learning Algorithms in Learning Management Systems

Emma Cheserem, Elizaphan Maina, John Kihoro, Jonathan Mwaura

► To cite this version:

Emma Cheserem, Elizaphan Maina, John Kihoro, Jonathan Mwaura. A Conceptual Framework for Automatic Generation of Examinations Using Machine Learning Algorithms in Learning Management Systems. IFIP World Conference on Computers in Education (WCCE), Aug 2022, Hiroshima, Japan. pp.441-450, 10.1007/978-3-031-43393-1_41. hal-05533304

HAL Id: hal-05533304

https://inria.hal.science/hal-05533304v1

Submitted on 2 Mar 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
<figure><img src="image_3.png" /></figure>

# A Conceptual Framework for Automatic Generation of Examinations Using Machine Learning Algorithms in Learning Management Systems

Emma Cheserem¹(✉) ID, Elizaphan Maina¹ ID, John Kihoro² ID,
and Jonathan Mwaura³ ID

¹ Kenyatta University, Nairobi, Kenya

{cheserem.emma,maina.elizaphan}@ku.ac.ke

² The Cooperative University of Kenya, Nairobi, Kenya

³ Northeastern University, Boston, MA, USA

j.mwaura@northeastern.edu

**Abstract.** The transition of education from face-to-face to electronic learning (e-learning) has been accompanied by the application of artificial intelligence and machine learning techniques to improve teaching, learning and assessment processes. Learning Management Systems (LMS) are used to conduct electronic learning (e-learning), and to facilitate student assessment through automatic generation of examinations from a question bank. However, the perceived low quality of these examinations has led them to be used for formative assessments and not for summative assessments. One way to ensure that high quality exams are generated by LMS systems would be to ensure that the questions cover different levels of difficulty as specified by an educational taxonomy. One commonly used taxonomy is Bloom's Taxonomy, later updated to the Revised Bloom's Taxonomy (RBT). In this research, we review studies on automatic generation of examinations from question banks. From this review, we define the parameters necessary for a quality exam based on RBT. Finally, we propose a conceptual framework that applies machine learning algorithms to automatically generate a quality exam from an LMS question bank. We intend to do further research by developing a prototype based on the conceptual framework.

**Keywords:** Automatic Generation of Exams · Machine Learning · Bloom's Taxonomy

## 1 Introduction

In recent decades there has been an accelerating shift from face-to-face learning to distance learning to electronic learning (e-learning), a change accelerated by the COVID-19 pandemic [1]. E-learning is made possible through Learning Management Systems (LMS) such as Moodle, Blackboard Learn and Canvas, tools that permit instructors
and learners to interact synchronously or asynchronously. As the shift to e-learning has occurred, machine learning (ML) techniques are being used to enhance the LMS systems in order to get better outcomes for instructors and learners [2, 3]. Studies show that applications of ML can improve learner engagement by providing individualised feedback to the student, provide rapid assessment and feedback to the student, enable intelligent agents to advise students, enhance administration and assessment in education, among many other applications [2, 4–6].

A normal teaching cycle requires an instructor to prepare several assessments, possibly including two continuous assessment tests (CATs), one regular examination and one supplementary examination, for each unit that s/he is teaching. This is a time-consuming and effortful process. One of the challenges faced, particularly in the setting of examinations, is ensuring that questions cover different levels of difficulty, to enable the instructor to determine the comprehension of students regarding the subject matter in a specific unit.

In LMS systems, examinations are generated by random selection of questions from a question bank, with no particular selection criteria [7–9]. Studies show that because it is expensive to generate a quality exam that tests different cognitive levels, online tests generated within LMS systems are primarily used for formative assessment, and not summative assessment [5]. However, research reveals that a quality examination should include questions that test both higher and lower order thinking skills. This can be done by ensuring that the questions in an examination cover all levels of an educational taxonomy such as Revised Bloom’s Taxonomy (RBT) [10]. RBT assists the lecturer in preparing intended learning outcomes (ILOs) and assessments [11]. While research has been done into coverage of RBT by examinations, most research does retrospective labelling of examination questions, not classification of questions prior to including them in examinations. However, using ML techniques to classify questions, it is possible to label questions in a question bank and thus enable generation of good quality examinations within an LMS.

The aim of this paper is to explore how machine learning algorithms can improve the quality of automatically generated exams in LMS systems. This study was guided by the following research questions:

1. What methods are used to automatically generate examinations from question banks in LMS systems?

2. What machine learning algorithms have been used to classify examination questions into different cognitive levels?

In this paper, we develop a conceptual framework to integrate ML techniques in the automatic generation of quality exams in LMS systems.

## 2 Methodology

For this review paper the Khan et al. [12] method was selected. It involves five steps: (i) framing research questions for a review, (ii) identifying relevant work, (iii) assessing the quality of studies, (iv) summarising the evidence, and (v) interpreting the findings. The review questions have been stated above. Subsequently, we identified primary studies
and systematic reviews using appropriate search keywords [13]. The following keywords were used:

* Automatic Examination Generation

* Online Test Generation

* Intelligent Test Paper Generation

* Automated Question Paper Generator System

The databases used for the search were Google Scholar and *IEEE Xplore*. The search period covered was 2011 - 2021. Out of this search, over 50 articles were selected, but scanning the abstracts allowed us to reduce the number to 10 papers relevant for this research, published between 2011 and 2021.

## 2.1 Examinations

The need to evaluate the learning attained by students is significant in Higher Education Institutions (HEIs). This is done through assessment tools such as quizzes, tests and final examinations. However, due to the significant time and effort required to set these assessments, some examinations are poorly designed, and fail to properly assess the achievement of the intended learning outcomes (ILO) of a specific course [14]. A good final examination should cover all the ILOs and have questions that can distinguish good from poor learners. RBT can be used to ensure questions of different difficulty level are included in the examination.

## 2.2 Question Bank

In an LMS system, a question bank is created when large numbers of individual questions of different types are drafted and placed in a repository. When an examination is required, questions are selected from the bank [14]. These questions may have undergone review and revision before they are added to the bank, or after being included in a test [15]. A good examination should contain questions from different levels of Revised Bloom's Taxonomy (RBT). We propose to use RBT to classify questions, prior to their inclusion in an examination generated from the test bank of an LMS.

An Automatic Examination Generator (AEG) is a tool that selects questions from a question bank and generates an examination. The AEG will use a pre-determined template to guide coverage of ILOs, randomize questions in order to avoid repetition and distribute marks fairly over the various questions in the exam [14].

Due to unfamiliarity with the process, or distrust of the quality of examinations produced, lecturers using LMS systems in HEIs use a hybrid of online and offline tools to conduct assessment [16–18]. To enhance the confidence of HEI examiners in automatically generated examinations, an AEG should produce a quality examination from the question bank, which tests different cognitive levels and covers the selected ILOs [19].
## 2.3 Bloom's Taxonomy

Educational taxonomies help the instructor to set learning objectives which then guide the learning activities and assessments [20]. One of the best known educational taxonomies is Bloom's taxonomy. It was developed in 1948 by a group of university lecturers attending the American Psychological Association Convention in Boston. Led by Benjamin Bloom, these educators collaboratively created a framework to classify learning outcomes [21]. They believed that this framework, named Bloom's Taxonomy, could promote standardisation in assessment and teaching [22]. Bloom's taxonomy had three components, the cognitive, the affective and the psychomotor domains. The cognitive component was widely adopted in education.

In 2001 the taxonomy was revised by a group of cognitive psychologists and assessment specialists including David Krathwohl, Lorin Anderson and other educational experts. They expanded Bloom's taxonomy from a single dimension to a two-dimensional taxonomy, updating the verbs and introducing a knowledge dimension [22]. In the Revised Bloom's Taxonomy (RBT), the categories in the cognitive dimension were renamed Remember, Understand, Apply, Analyse, Evaluate and Create as shown in Fig. 1 below [23].

<figure><img src="image_4.png" /><figcaption>Fig. 1. Revised Blooms Taxonomy</figcaption></figure>

## 2.4 Use of ML Techniques to Classify Examination Questions

ML techniques are used widely to classify examination questions. A review of the literature showed several popular ML techniques, including Support Vector Machines (SVM), K-Nearest Neighbour (kNN), Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM), Naïve Bayes, Logistic Regression, Decision Trees and Random Forest used to classify questions into a level of RBT. Of these ML techniques, SVM was one of the most popular, attaining 86% correct classification as seen in the analysis of results below.
# 3 Results

In this section we critique studies on the automatic generation of examinations with an emphasis on how questions are selected for inclusion in an examination. A good examination covers the ILOs of a course, and contains questions that range over the different levels of RBT. This means that the questions test both higher and lower order thinking skills, enabling the examiner to use the examination as a tool to discriminate between students who have a shallow understanding and those who have a deep understanding of the subject. Such an examination should also align with the question paper format desired by the examiner [24].

Kale and Kiwelekar [24] designed a AEG that enables the examiner to choose the examination questions according to learning outcome, the cognitive skill level of RBT to be covered, the difficulty level, and the desired marks per question. The algorithm then selects questions from the question bank as long as they meet the desired criteria. This question paper generator depends on the examiner to correctly specify the RBT level of the question, and it does not include any ML techniques.

Amria et al. [14] present a framework that allows educators to map questions to ILOs based on RBT. In their system, the educator is responsible for defining a question and mapping it to a predefined ILO. Questions that are mapped to a lower level of RBT such as remembering, understanding, and applying are placed in the first part of the exam while those at a higher RBT level are placed in the latter part of the exam. Their tool allows tutors to enter different question types such as True/False, essay questions and others. Output formats include XML (for electronic exams), PDF, or Document. This tool uses a random algorithm to select questions to be included in the exam, while categorization based on RBT is done by the examiner when adding questions to the question bank.

Several of the papers we found showed the steps taken to classify examinations according to RBT. In a 2016 study, Jayakodi et al. [25] used Natural Language Processing (NLP) techniques to identify the RBT category for exam questions from the Computer Science domain. Using NLP to develop a set of rules for classification, their system was able to correctly categorise 82% of 147 questions that had been previously labelled by a domain expert. The study could be extended to cover a broader range of questions from different educational domains.

A similar study by [26] used a rule-based approach and NLP to identify the keywords in a question, and subsequently categorized a set of 100 questions into the different RBT levels. The study emphasised that a certain keyword in a question does not automatically determine the RBT category. Their research calculated the question category based on weights calculated by subject matter experts (SMEs).

Other studies in our review classified the questions in examinations into RBT using ML techniques. Laddha et al. [27] investigate the use of CNN and LSTM to classify 844 exam questions into RBT levels. Using a dataset labelled with both cognitive and knowledge dimensions, they trained the CNN and LSTM models on a 70% training set and tested on the remaining 30%. The limitation of this study was the low accuracy levels of 66.67% for CNN, and 44% for LSTM.

An empirical research by [11] used a Python script to match the action word in the examination question to a RBT keyword list. Using a dataset of 1,000 examination
questions, they then used an open source tool named orange and the machine learning algorithms SVM and kNN to classify the questions into the 6 different RBT levels. The accuracy for SVM was 69% and for kNN was 60.5%.

Mohammed & Omar [10] use term frequency-inverse document frequency (TF-IDF) and word2vec to improve the performance of exam question classification according to Bloom's Taxonomy. In their research, they evaluate the performance of SVM, Logistic Regression (LR), and kNN. Their findings were that SVM had superior performance over LR and kNN, achieving an average of 89.7%, 89.4%, and 85.44% weighted F1-measure respectively. Their research classified questions from two datasets containing a total of 741 questions. They found ML techniques useful in classifying examination questions from different domains.

The research done by Rahim et al. [7] takes a database of 500 examination questions and uses the Genetic Algorithm to select the questions to be included according to coverage of RBT levels. This research describes a good examination as that which covers 3 to 6 levels of RBT. Although this research was restricted by the quantity of questions they had available for their research, they were able to generate examinations that had on average 70% coverage of RBT.

Sangodiah et al. [28] investigated the performance of the SVM algorithm in classifying questions using different taxonomy-based features. The questions in the study were classified into RBT levels by domain experts. The questions were pre-processed using WordNet, while Stanford Parser was used to tag parts-of-speech. The output was then passed through a feature extraction process and the extracted features used to train the SVM classifier. The findings were that bag-of-words combined with general and specific taxonomy terms were the most useful features in question classification. The classifier was able to achieve 72.9% accuracy. A summary of the ML techniques used in examination question classification is shown in Table 1 below.

## 3.1 Proposed Conceptual Framework for the Automatic Generation of Exams

In this section we propose a machine learning enabled conceptual framework for the automatic generation of examinations from a question bank in an LMS. The variables in this framework include coverage of ILOs, coverage of RBT, marks distribution, and random selection of question [7, 14]. The questions used in the automatic generation of exams will originate from a question bank populated by different types of questions in a specific subject domain. The question bank can be created by manual input of questions by the instructors; alternatively questions may be generated using artificial intelligence techniques including NLP [10, 29].

From the studies reviewed, it is evident that a number of researchers are using NLP techniques and ML algorithms to classify examination questions into the different RBT levels. Concurrently, several researchers have developed AEGs that produce examinations; however these use random selection of questions that fit predetermined criteria.

Based on the above reviews, we propose to improve AEGs by using ML techniques to label the questions with the correct RBT levels prior to their selection for inclusion in an examination. The performance of the ML model would improve over time as questions are added to the question bank. We opine that the use of ML techniques would result
<table><caption>**Table 1.** Summary of ML Techniques used in classifying examination questions</caption><thead><tr><th></th><th>Authors</th><th>ML Algorithm (Accuracy)</th><th>Question datasets used</th></tr></thead><tbody><tr><td>1</td><td>Laddha et al. [27]</td><td><ul><li>CNN (80%)</li><li>LSTM (71%)</li></ul></td><td>A dataset of 844 questions</td></tr><tr><td>2</td><td>Patil & Shreyas [11]</td><td><ul><li>Used SVM and kNN to classify the questions into 6 different levels</li><li>SVM (69%)</li><li>kNN (60.5%)</li></ul></td><td>The data set consisted of 1,000 questions collected from various external sources related to operating system course</td></tr><tr><td>3</td><td>Mohammed & Omar [10]</td><td><ul><li>Support Vector Machine (SVM) (89.7%)</li><li>Logistic Regression (LR) (89.4%)</li><li>K-Nearest Neighbour (kNN) (85.4%)</li></ul></td><td>Dataset 1 – 141 questions<br>Dataset 2 – 600 questions</td></tr><tr><td>4</td><td>Rahim [7]</td><td><ul><li>Genetic Algorithm</li><li>Exams generated had 70% coverage of RBT</li></ul></td><td>A dataset of 500 questions</td></tr><tr><td>5</td><td>Sangodiah et al. [28]</td><td><ul><li>Domain experts</li><li>SVM (72.9% accuracy)</li></ul></td><td>A dataset of 415 questions</td></tr></tbody></table>

in good quality examinations that can be used for generating summative exams from LMS systems. The diagram below illustrates how the proposed ML-enabled AEG would work.

The conceptual framework below envisions examination questions entered into the question bank, then using a trained ML model, the questions are then classified into different RBT levels. Subsequently, an examination can be automatically generated that adheres to a defined question paper format, comprising questions covering the ILOs as prescribed by the examiner and covering all the RBT levels (Fig. 2).
<figure><img src="image_5.png" /><figcaption>Fig. 2. Proposed Conceptual Framework</figcaption></figure>

## 4 Conclusion

This study reviewed a number of studies on the automatic generation of examinations using machine learning techniques. From the review, we have defined a number of parameters necessary for the automatic generation of examinations using machine learning algorithms to label the questions in a question bank. Having noted the need to generate good examinations, we propose a conceptual framework that will utilise a trained machine learning model to label the questions prior to their inclusion in a quality exam. We propose to validate this framework by developing a prototype based on the conceptual framework. Using this prototype, examinations will be produced for a specific subject domain in an LMS system.

**Acknowledgement.** This research was supported by the National Research Fund 2016/2017 grant award under the multidisciplinary-multi-institutional category involving Kenyatta University, University of Nairobi, and The Cooperative University of Kenya. The research is investigating how artificial intelligence can be used to enhance e-learning in HEIs.

## References

1. Schiff, D.: Out of the laboratory and into the classroom: the future of artificial intelligence in education. AI Soc. **36**(1), 331–348 (2020). https://doi.org/10.1007/s00146-020-01033-8

2. Popenici, S.A.D., Kerr, S.: Exploring the impact of artificial intelligence on teaching and learning in higher education. Res. Pract. Technol. Enhanc. Learn. **12** (2017)

3. Zawacki-Richter, O., Marin, V.I., Bond, M., Gouverneur, F.: Systematic review of research on artificial intelligence applications in higher education – where are the educators? Int. J. Educ. Technol. High. Educ. **16** (2019)

4. Araka, E., Maina, E., Gitonga, R., Oboko, R.: Research trends in measurement and intervention tools for self-regulated learning for e-learning environments-systematic review (2008–2018). Res. Pract. Technol. Enhanc. Learn. **15** (2020)
5. Boitshwarelo, B., Reedy, A.K., Billany, T.: Envisioning the use of online tests in assessing twenty-first century learning: a literature review. Res. Pract. Technol. Enhanc. Learn. **12** (2017)

6. Luckin, R.: Towards artificial intelligence-based assessment systems. Nat. Hum. Behav. **1** (2017)

7. Rahim, T.N.T.A., Aziz, Z.A., Rauf, R.H.A., Shamsudin, N.: Automated exam question generator using genetic algorithm. In: 2017 IEEE Conference on e-Learning, e-Management and e-Services, IC3e 2017, pp. 12–17. IEEE (2017)

8. Kalaluka, K.M.: Exam paper generating system: automated exam system. Int. J. Multi-Discip. Res. 1–4 (2017)

9. Noor, N.M., Napi, N.M., Amin, I.F.I.: The development of autonomous examination paper application: a case study in UiTM Perlis branch. J. Comput. Res. Innov. (JCRINN) **4**, 21–30 (2019)

10. Mohammed, M., Omar, N.: Question classification based on Bloom's taxonomy cognitive domain using modified TF-IDF and word2vec. PLoS ONE **15**, 1–21 (2020)

11. Patil, S.K., Shreyas, M.M.: A comparative study of question bank classification based on revised bloom's taxonomy using SVM and K-NN. In: 2nd International Conference on Emerging Computation and Information Technologies (ICECIT), pp. 1–7. IEEE (2017)

12. Khan, K.S., Kunz, R., Kleijnen, J., Antes, G.: Five steps to conducting a systematic review. J. Roy. Soc. Med. **96** (2003)

13. Kitchenham, B.: Guidelines for performing Systematic Literature Reviews in Software Engineering Version 2.3 EBSE Technical Report EBSE-2007-01 (2007)

14. Amria, A., Ewais, A., Hodrob, R.: A framework for automatic exam generation based on intended learning outcomes. In: Proceedings of the 10th International Conference on Computer Supported Education (CSEDU 2018), pp. 474–480 (2018)

15. Crisp, V., Shaw, S., Bramley, T.: Should we be banking on it? Exploring potential issues in the use of “item” banking with structured examination questions. Asses. Educ.: Princ. Policy Pract. **27**, 655–669 (2020)

16. Mimirinis, M.: Qualitative differences in academics' conceptions of e-assessment. Assess. Eval. High. Educ. **44**, 233–248 (2019)

17. Rolim, C., Isaias, P.: Examining the use of e-assessment in higher education: teachers and students' viewpoints. Br. J. Educ. Technol. **50**, 1785–1800 (2018)

18. Ivanova, M., Bhattacharjee, S., Marcel, S., Rozeva, A.: Enhancing Trust in eAssessment - The TeSLA System Solution. Technical University of Sofia (2019)

19. Tetali, D.R., Rani, P.K.: Automated course outcomes assessment for multiple choice questions (auto_assess). Int. J. Adv. Res. Comput. Sci. **8**, 189–192 (2017)

20. Van Niekerk, J., von Solms, R.: Using Bloom's taxonomy for information security education. In: Dodge, R.C., Futcher, L. (eds.) WISE 2009/2011/2013. IAICT, vol. 406, pp. 280–287. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-39377-8_33

21. Bloom, B.S., Engelhart, M.D., Furst, E.J., Hill, W.H., Krathwohl, D.R.: Taxonomy of Educational Objectives: The Classification of Educational Goals. Taxonomy of Educational Objectives, p. 207 (1956)

22. Anderson, L.W., et al.: A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives (2001)

23. Krathwohl, D.R.: A revision of Bloom's taxonomy: an overview. Theory Pract. **41**, 212–218 (2002)

24. Kale, V.M., Kiwelekar, A.W.: An algorithm for question paper template generation in question paper generation system. In: The International Conference on Technological Advances in Electrical, Electronics and Computer Engineering (TAEECE), pp. 256–261. IEEE (2013)
25. Jayakodi, K., Bandara, M., Perera, I.: An automatic classifier for exam questions in engineering: a process for Bloom's taxonomy. In: Proceedings of 2015 IEEE International Conference on Teaching, Assessment and Learning for Engineering, TALE 2015, pp. 195-202 (2016)

26. Omar, N., et al.: Automated analysis of exam questions according to Bloom's taxonomy. Proc. - Soc. Behav. Sci. **59**, 297-303 (2012). 8

27. Laddha, M.D., Lokare, V.T., Kiwelekar, A.W., Netak, L.D.: Classifications of the summative assessment for revised Bloom's taxonomy by using deep learning. Int. J. Eng. Trends Technol. **69**, 211-218 (2021)

28. Sangodiah, A., Ahmad, R., Ahmad, W.F.W.: Taxonomy based features in question classification using support vector machine. J. Theor. Appl. Inf. Technol. **95**, 2814-2823 (2017)

29. Kumar, V., Ramakrishnan, G., Li, Y.-F.: A framework for question generation from text. In: 2019 IJCAI Workshop SCAI: The 4th International Workshop on Search-Oriented Conversational AI (2019)