# TRIZ-Based Remodeling of Multiple True-False Questions

Ashley Soosay, Narayanan Kulathuramaiyer

► To cite this version:

Ashley Soosay, Narayanan Kulathuramaiyer. TRIZ-Based Remodeling of Multiple True-False Questions. TRIZ Future conference, Sep 2021, Bolzano, Italy. pp.355-366, 10.1007/978-3-030-86614-3_28 hal-04067809

HAL Id: hal-04067809

https://inria.hal.science/hal-04067809

Submitted on 13 Apr 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# TRIZ-based Remodeling of Multiple True-False Questions

Ashley Edward Roy Soosay¹ and Narayanan Kulathuramaiyer ²

¹ Faculty of Medicine and Health Sciences, 94300 Kota Samarahan, Universiti Malaysia Sarawak, Sarawak, Malaysia

² Institute of Social Informatics and Technological Innovations, 94300 Kota Samarahan, Universiti Malaysia Sarawak, Sarawak, Malaysia

sashley@unimas.my

**Abstract.** In light of the COVID-19, medical education is widely transitioning to a fully online mode. This change poses challenges on ensuring valid assessments for medical education. Within the constraints of online assessment scenario, the academic rigour has to be maintained, while at the same time, enabling a remote administration of student learning activities. A TRIZ-based approach has been adopted in addressing this contradiction. We used TRIZ tools such as Functional Analysis, Cause and Effect Chain Analysis, Engineering and Physical contradiction to model the problem. The TRIZ knowledge models, particularly through the recommended Inventive Principles has provided insightful directions as research ideas. This paper then explored a combined validation method based on the assessment activities conducted in the medical faculty during the COVID-19 lockdown. The hybridized TRIZ mode has served as an excellent knowledge elicitation framework for addressing this long-standing problem.

**Keywords:** Multiple True-False Question, TRIZ, Knowledge-based modelling

## 1 Introduction

The COVID-19 pandemic has been the impetus for medical education in preclinical years going totally online. The declaration of COVID-19 as a worldwide pandemic [1] has led to the Movement Control Order (MCO) in Malaysia [2-5], which in turn has forced institutions of learning to adopt and adapt new norm in teaching and learning (T&L) process. The COVID-19 outbreak not only challenged the learners but also the T&L service providers to switch from face to face (F2F) classroom settings to remote learning (RL) [6, 7]. The mental well-being of people worldwide is affected by this pandemic [8]. The outbreak has thus resulted an unparalleled instructional environment.

Many institutions have provided internet access to their students and lecturers in an attempt to facilitate a smooth transition to the new pedagogy which largely relies on the quality of internet connectivity [9]. Mastering the use of online communication platform has been a pivotal factor to ensure success in the design of an engaging RL. In Universiti Malaysia Sarawak (UNIMAS), T&L is delivered electronically via its online learning management system, e-Learning Enrichment and Advanced Platform (e-LEAP), which is based on Moodle platform. Synchronous and asynchronous T&L is being conducted with recorded lectures, remote problem based learning (PBL) sessions and clinical video sessions serving as the pedagogy of RL.
What comes hand in hand with T&L is assessment. With such constraints in administering F2F T&L delivery, online assessment becomes the primary mode of examining students' T&L. Online assessment has therefore taken a paramount role in driving learning, during this challenging period [10]. The shift from paper-based learning to an online mode has been a drastic change for the Faculty of Medicine & Health Sciences, UNIMAS. The paper-based-assessment which comprised of Multiple Choice Question (True/False) (MTFQ), Best Answer Question (BAQ), Objective Structured Practical Examination (OSPE) and Modified Essay Question (MEQ), was required to be replicated using the eLEAP platform, with reassessment considerations.

The use of MTFQ has been seen suitable for assessing the first three cognitive levels of remembering, comprehending, applying and to a great extent the level of analysis [11, 12]. At the same time addressing the high failure rates associated with the MTFQ mode of assessment is pertinent. As the ideal choice of assessment methods are still a subject of debate in the academic scenario, this paper presents an insightful dissection of this problem via Theory of Inventive Problem Solving (TRIZ). This paper explores a TRIZ-based approach to model this long-standing problem, in shedding insights on current methods being used as well as to reveal directions for potential innovative solutions. Moreover, the COVID-19 pandemic has presented an opportunity to further examine this modality in our assessment tool using the e-LEAP platform. In this paper we investigate using a case study to achieve an ideal (or innovative) form of assessment suited for medical students.

TRIZ, which has been founded by Genrich Altshuler in the former Soviet Union, has been widely used as a means of solving complex problems [13]. TRIZ was explored as a problem-solving methodology due to its strengths in dealing with complex problems [14] such as high failure rates in MTFQ assessment among preclinical students. The research project was initially explored as a challenge in a TRIZ training. The main author with over 25 years of experience as a medical educator, was then intrigued by the simplistic Engineering systems model and the analogical reasoning capability that it provided.

The use of tools such as Cause and Effect Chain Analysis (CECA) [15], Function Models and Physical Contradictions were explored to shed insights to this long-standing problem. TRIZ has therefore provided the necessary visual modelling tools in assisting to deal with an ill-structured problem.

## 2 Research methodology

The modelling of online learning delivery was adopted as a case study, considering the pandemic situation and the forced online evaluation. While at the same time examining a more effective mode for shaping future medical education.

Firstly, we reviewed the literature encompassing areas of medical education assessment modalities and then probing the causes for high failure rates in MTFQ. The medical scenario that shaped the research was outlined further and followed by preliminary analysis using Function Analysis and Cause and Effect Chain Analysis. Interviews were then conducted with experts in the field of medical education to validate the logical flow in the cause-effect relationships. The preliminary TRIZ models that
guided to contextualize the research was apparent. Table 1 shows the overall concept of this project's methodology.

<table><caption>**Table 1.** Overall concept methodology of the project.</caption><thead><tr><td>Research Methodology</td><td>Tools</td></tr></thead><tbody><tr><td>Review of Related Works</td><td>Review covered areas of Single Answers vs Multiple True False modes of assessments and Study of stressors in medical education.</td></tr><tr><td>Interview with Domain Expert</td><td>CECA was used as a visual tool to generate and test hypothesis on the pathways to solve problem.</td></tr><tr><td>Problem Modelling based on Function Analysis</td><td>Function Model was used to model domain knowledge based on focus group discussions.</td></tr><tr><td>Systems Analysis as a process to formulate inventive problem solving model</td><td>Engineering Contradiction as a knowledge-based approach was used to identify directions.</td></tr><tr><td>Contradictory Analysis as a deep knowledge model of inventive solution</td><td>Physical Contractions then served as an instrument for deepening inquiry.</td></tr><tr><td>Evaluation of Hypothesis</td><td>Review partial solutions was performed with an engagement with stakeholders to qualitatively enhance visual domain models.</td></tr><tr><td>Analysing Student Performance in Summative Assessment</td><td>MTFQ modality examination component was analysed to look for emerging patterns.</td></tr></tbody></table>

The approach has demonstrated the potential application of these TRIZ tools as a means for knowledge elicitation by interacting with experts to construct knowledge expert models. Subsequently Engineering Contradiction and Physical Contradiction were employed as analytical tools for modelling the problem in an entire new way, shedding insights on the problem at hand. Each of these tools presented a series of inventive principles by applying heuristic approach.

The recommended solutions were mapped and directions for achieving these concepts were contrasted with features within the existing eLEAP platform. The feature analysis provided alternative ways of modelling the modalities. The exploratory approach that employs the series of TRIZ tools is illustrated in Table 1.

## 2.1 Medical Examination in UNIMAS Case Study

During an undergraduate medical training, the clinical practice of prevention, health and wellness is inculcated to future practicing doctors. The difficulty of this training in reality has to deal with a variety of stressors as experienced by the trainees [16-18]. Vyas et. al., reported that there is a significant level of stress among the female preclinical year students in Southeastern US allopathic medical school. Garber, et. al., reported that stress levels among pharmacy students were significantly higher compared to standard populations. Pressure to succeed is one of the strong predictor of perceived stress among students. Hundertmark, et. al., stated that students who generally exhibit
a high general stress levels in medical school are subjected to stressors such as time pressure, participant characteristics, teacher-role demands and study requirement.

To illicit the nature of high failure rate of students in the MTFQ modality examination, a focus group approach consisting of experts in the area of Medical Education, Cognitive Science, Knowledge Engineer Medical Expert, Educational Pedagogy and Clinical Psychologist Specialist were utilized. From the interview with Clinical Psychologist a Cause and Effect Chain analysis was formulated.

## 2.2 TRIZ-based Hybrid Methodology

Students' ability to perform across modalities was studied based on targeted student groups. Initial analysis revealed that students under performed in MTFQ modality examinations. Further to this the focus group session highlighted that stress was perceived as a key factor in students' inability to perform to their fullest potential. Interviews with students revealed that, not being prepared and being overwhelmed by diverse assessment needs were the stressors that affected their performance.

One of the key findings was that students tend to lack the ability to map their level of knowledge mastery into a successful answering strategy. Upon discussions with students who did well in the examination, it became apparent that a strategy to balance a risk taking attitude with an adequate field knowledge mastery was the key factor. Although the beneficial aspect of MTFQ compared to other modalities has not been extensively studied, Karimi and Manteufel showed that electronic administration of True/False multiple choice questions helps in measuring and reinforcing the understanding of fundamental concepts in Thermal Science course [19]. In the analysis of test items, the likelihood of right answer being correct can also be easily mapped to an answering strategy. The mapping of False statement into an answering strategy required additional cognitive loads owing to its added element of risk.

## 2.3 Function Modelling

<figure><img src="image_3.png" /><figcaption>Fig. 1. Function model.</figcaption></figure>

As a problem modelling step, the main useful function of the MTFQ modality is presented in Figure 1. Although the initial model formulated the examination modality, with components such as Stem, Option, Answer, Distractor, Penalty, it became evident that a cognitive process modelling was crucial in highlighting variations in quality of assessments (see Table 2). We then came up with the perceptual cognitive mapping as shown in Figure 2 which took into considerations the lessons learnt in the TRIZ-based knowledge representation stage for solving such a complex model.
<table><caption>**Table 2:** Preliminary Component Analysis.</caption><thead><tr><th>Components</th><th>System</th><th>Product</th><th>Super system (Cognitive model)</th></tr></thead><tbody><tr><td>Items</td><td>Question Stem</td><td>Learner</td><td>Knowledge-level indicator</td></tr><tr><td></td><td>Option</td><td></td><td>Stress level indicator</td></tr><tr><td></td><td>Answer</td><td></td><td>Fear of Wrong Answer</td></tr><tr><td></td><td>Distractor</td><td></td><td>Willingness to take risk</td></tr><tr><td></td><td>Penalty</td><td></td><td>Modality</td></tr><tr><td></td><td>Medium</td><td></td><td>True/False decision</td></tr></tbody></table>

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Cause and Effect Chain Analysis</figcaption></figure>

The knowledge base representation shown in Figure 2 was acquired through discussions with a Clinical Psychologist based on an iterative collaborative Cause and Effect Chain Analysis (CECA) modelling. Cause and Effect Chain Analysis enabled the flow of questioning leading to the potential hypothesis to direct the second stage of enquiry to the model of interdependencies. Three major components were identified based on interviews with the Clinical Psychologist in the faculty who has a deep knowledge in students' capabilities in their examinations. The factors in the lower end of the CECA diagram were directly gauged from the interview with students who had difficulty in academic performance. Furthermore, a detail analysis of the right side of the CECA diagram allowed us to apply Function Analysis, to further study the overall function of the technical aspects of the MTFQ modality examination. The function analysis diagram is shown in Figure 3. The initial concepts gained from interviews with students served as triggers in the construction of this model.
<figure><img src="image_5.png" /><figcaption>Fig. 3. Function Analysis of Answer Strategy</figcaption></figure>

The heuristic models (Figure 2 and 3) highlighted the underlying cause together with the interdependencies of the elements concerned. It allowed a visualization of observed effects via a graphical way. Based on this model, the cause for the high failure rate among preclinical students in MTFQ modality examination was evaluated. In using self-directed learning method, the cognitive loads of students were observed to be much higher when they identify a False statement, as compared to identifying a True statement. Awareness of a factual knowledge as evidence is sufficient to allow student to pinpoint True statements, where else False statements is based on “being convinced to the best of one’s knowledge”. Gathering factual knowledge as evidence for True statements requires less cognitive load on students. As a result, identifying of modalities requiring the False statement is a likely cause that posed as an added stressor. There was a need to investigate further if the negative question together with a need to context switching was the main cause for the high failure rates. TRIZ was thereby used as knowledge modelling framework for formulating these challenge scenarios.

## 2.4 Engineering Contradictions

Based on the initial analysis, the following engineering system was formulated. **IF MTFQ is used, THEN learner knowledge is tested comprehensively BUT learners may experience high stress or pressure to perform.** The improving parameter was selected as “productive” indicating the enabling level of learners’ to attempt MTFQ in a productive manner. Meanwhile the worsening parameter was selected as “Stress or Pressure”, which indicates the learners feeling of stress level while attempting the MTFQ.
The Inventive principles then provided insights in the modelling of MTFQ assessments by suggesting intuitive strategic directions. Based on these directions we have formulated the possible approaches as illustrated in Table 3.

<table><caption>**Table 3.** Engineering Contradiction based on MTFQ modality used in examination with considering the improving parameter as Productive and the worsening parameter as Stress to the Learners was considered and the possible approaches postulated.</caption><thead><tr><td>Inventive Principle</td><td>Possible approach</td></tr></thead><tbody><tr><td>Preliminary Action</td><td>Prepare the student beforehand on the usage of the examination modality. This approach was in practise by the instructors and was also recommended by resolving this Engineering Contradiction.</td></tr><tr><td>Phase Transitions</td><td>To introduce interventions so that stressors are systematically harmonised so that the impact of the stressors will be neutralised.</td></tr><tr><td>Curvature</td><td>Students to apply various strategy while attempting the exam modality to simulate the sense of circularity (revisiting the questions based on certain algorithm). These strategies can be inculcated to the students to do.</td></tr></tbody></table>

## 2.5 Physical Contraditions

Finally, in utilizing Physical Contradiction, a contradictory analysis as a deep model of inventive solution (Table 4) was acquired. Table 4 highlights the reasoning strategies based on the three inventive principles identified via Physical Contradiction.

<table><caption>**Table 4.** Based on the recommendation from Engineering and Physical contradictions, an evaluation was carried out to determine the effectiveness of the proposed strategy. We next contrasted the performance of students before and after the intervention.</caption><thead><tr><td>Inventive Principle</td><td>Recommendation on Remodelling</td></tr></thead><tbody><tr><td>Segmentation</td><td>Need to structure questions or groups (segments) for reduced stress with a quicker answering strategy. This approach relates particularly to students deciding on which questions to answer first, and they mark questions for a second round analysis. In this context, training could be provided prior to the exams on how to segment the questions based on "ability to respond"</td></tr><tr><td>Homogeneity</td><td>Use the same medium as interface for a more focused answering strategy for students (stress reduced). We explore the possibility of using the same question form, but as a stress reducer, students only identify True statements. They are not required to ascertain the truth value of distractors.</td></tr><tr><td>Equipotentiality</td><td>Reduce the need for learners to determine the truth value of every statement. Identifying True statements is considered adequate.</td></tr></tbody></table>
# 3 Insights from the TRIZ-based models

<figure><img src="image_6.png" /><figcaption>**Fig 4.** The clustered bar chart shows the result in % for passing rate in MTFQ modality examination among preclinical students for 3 different courses, namely MDP10108, MDP10408 and MDP10507. The vertical downward black arrow indicates the beginning of the COVID-19 Pandemic, where the implementation of online examination via the eLEAP platform.</figcaption></figure>

Figure 4 shows a sudden increase in passing rate in MTFQ modality examination among preclinical students in MDP10108, MDP10408 and MDP10507 courses after March 2020 (indicated by the vertical downward black arrow in Figure 4). The COVID-19 pandemic movement control order (MCO) was imposed in March 2020. The difference in the assessment of MTFQ pre and post pandemic (Table 5 Type A vs. Type B) has been mainly in the use of eLEAP platform. A MTFQ consist of a stem and five options. The author of the MTFQ decides an option to be True or False in any sequence order. Therefore, any number of options within 0 to 5 can be True or False.

Prior to MCO, paper based assessment of MTFQ required a student to indicate the choice of option, which can be True, False or don't know (omit) by using an optical mark reader (OMR) sheet. Upon completing, the OMR is then scored using SmartScan software. One mark is awarded for correct answers while an incorrect answer is penalized by deducting one mark. An omitted option does not carry any value. Therefore, this enables a maximum mark awarded for a MTFQ to be 5 marks and the minimum mark is zero. The minimum marks are normalised to zero in order not to carry forward negative marks to the next question.
<table><caption>**Table 5:** The table shows the description of a question with the respective answer given in Answer Column (shaded). Type A is the pre pandemic version (paper based assessment F2F) while type B is the post pandemic (via eLEAP online assessment). Student responses 1-4 are hypothetical answers from 4 different students. The results row shows the marks awarded for all the situation arising from SR-1 to SR-4.</caption><thead><tr><th>A: The following colours can be found in the rainbow,</th><th>Answer</th><th>SR-1</th><th>SR-2</th><th>SR-3</th><th>SR-4</th></tr></thead><tbody><tr><td>A. blue</td><td>T</td><td>T</td><td>T</td><td>T</td><td>T</td></tr><tr><td>B. green</td><td>T</td><td>T</td><td>T</td><td>T</td><td>T</td></tr><tr><td>C. black</td><td>F</td><td>F</td><td>F</td><td>T</td><td>F</td></tr><tr><td>D. brown</td><td>F</td><td>F</td><td>T</td><td>T</td><td>F</td></tr><tr><td>E. red</td><td>T</td><td>F</td><td>F</td><td>F</td><td></td></tr><tr><td>*RESULTS:</td><td>5</td><td>3</td><td>2</td><td>0</td><td>4</td></tr></tbody></table><p>* penalty for wrong answer (minus 1)</p><p>SR - student response</p><table><thead><tr><th>B: The following colours can be found in the rainbow,</th><th>Answer</th><th>SR 1</th><th>SR 2</th><th>SR 3</th><th>SR 4</th></tr></thead><tbody><tr><td>A. blue</td><td>T</td><td>T</td><td>T</td><td>T</td><td></td></tr><tr><td>B. green</td><td>T</td><td>T</td><td>T</td><td>T</td><td>T</td></tr><tr><td>C. black</td><td></td><td></td><td>T</td><td>T</td><td>T</td></tr><tr><td>D. brown</td><td></td><td></td><td></td><td>T</td><td>T</td></tr><tr><td>E. red</td><td>T</td><td>T</td><td>T</td><td>T</td><td>T</td></tr><tr><td>*RESULTS:</td><td>5</td><td>3.33</td><td>3.33</td><td>1.66</td><td>0</td></tr></tbody></table>

SR - student response

During the pandemic, F2F interaction has been minimised and remote learning is engaged actively. The MTFQ is then replicated via eLEAP (learning management system). The MTFQ still has a stem and five options and any number of options within 0 to 5 can be True or False. The only difference is the choice of answers selected by the students. The students are required only to identify True options and key in their selection online. The marks for each question varies depending on the number of True options. The different marking scheme is depicted in Table 5.

## 4 Discussion

TRIZ-based models enabled us to tackle the aged problem in the field of medical education assessment. The modality of MTFQ has a high fatality rate. Applying problem solving tools available in TRIZ allowed us to deduce a solution to the pressing problem. Stress was identified as one of the causes of this problem together with some cognitive and technical aspects. Function Analysis indicated the elements that may be influencing

<footnote>* penalty for wrong answer (minus depends on number of correct options), in this example it is 1.66 %</footnote>
the goal of achieving high productivity among the learners. Engineering contradiction of “IF MTFQ is used, THEN learner knowledge is tested comprehensively BUT learners may experience high stress or pressure to perform” led us to deduce “productivity” as improving parameter while the worsening parameter was attributed to “stress”.

This in return enabled us to gain inventive principles such as preliminary action and curvature. Preparing learners prior to assessment may provide high productivity. Whereas learners can be coached to use various strategy mimicking a sense of circulatory such as revisiting questions based on refined algorithm. Through the use of physical contradiction, the inventive principles such as segmentation, homogeneity and equipotentiality came into light. Segmenting questions on the ability to respond is an example of this invention. As for equipotentiality, identifying only True statements instead of True and False can be attributed. The last inventive principle homogeneity has the ability to minimize stress through the application reducing the cognitive load. Learners are required to seek True statement only and this may contribute a homogeneous thought process while attempting MTFQ.

# 5 Conclusion

TRIZ-based models which has been adapted for engineering domains has been shown in this paper to serve as a guide to move solution towards an ideal pathway through its heuristic modelling. The tools used in this project were incrementally improved and suited for the context of the discipline in query. This shows that TRIZ has the ability to produce a customized solution. TRIZ has act as a catalyst for remodeling of an ill structured multi-disciplinary real-world problem, with the goal of moving towards an ideal solution.

The exploration has demonstrated the potential use of TRIZ tools as aid for knowledge elicitation to help in modelling complex problems. TRIZ-based models can be used in enabling a structural modelling of future medical examination modality. The pandemic situation has turned out to be a blessing in disguise as it provided an opportunity for a methodical research/inquiry into the long-standing problem. Future studies may scrutinize the currently used model of assessment in post pandemic situation when F2F interaction is applicable. This will allow a comparison of pre and post using the current model of assessment.

The evaluation carried out revealed that the TRIZ based partial solutions were in fact workable. The preliminary findings demonstrated that future assessments can be formulated to serve as an ideal examination modality able to minimize stress effectively. Though interesting directions have been highlighted by the heuristic models discovered in this paper further research would be needed to test out and validate this hypothetical finding.

**Acknowledgments.** The authors wish to thank Centre for Applied Learning and Multimedia, Universiti Malaysia Sarawak for the eLEAP support for this project.
References

1. Bahaeldin, H., Ayed A. Shati, Abdulaziz Alamri, Ayyub Patel, Ali Alsuheel Asseri, Muhammed Abid, Saleh M. Al-Qahatani and Ismaeel Satti, *Online assessment for the final year medical students during COVID-19 pandemics; the exam quality and students' performance*. Oncology and Radiotherapy, 2020. **14**(6): p. 1-6.

2. Bunyan, J., PM: Malaysia under movement control order from Wed until March 31. *all shops closed except for essential services*, in *Malay Mail*. 2020, Malay Mail: Kuala Lumpur.

3. Jun, S.W., *Movement control order not a lockdown, says former health minister.* , in *Malay Mail*. 2020, Malay Mail: Kuala Lumpur.

4. Sukumaran, T., *Malaysia in partial lockdown to limit coronavirus spread*., in *South China Morning Post*. 2020, South China Morning Post.: Kuala Lumpur.

5. Tang, K.H.D., *Movement control as an effective measure against Covid-19 spread in Malaysia: an overview*. Zeitschrift fur Gesundheitswissenschaften = Journal of public health, 2020: p. 1-4.

6. Ahmed, H., M. Allaf, and H. Elghazaly, *COVID-19 and medical education*. The Lancet. Infectious diseases, 2020. **20**(7): p. 777-778.

7. Alsafi, Z., et al., *The coronavirus (COVID-19) pandemic: Adaptations in medical education*. Int J Surg, 2020. **78**: p. 64-65.

8. Wang, C., et al., *The impact of COVID-19 pandemic on physical and mental health of Asians: A study of seven middle-income countries in Asia*. PLoS One, 2021. **16**(2): e0246824.

9. Serhan, D., *Transitioning from Face-to-Face to Remote Learning: Students' Attitudes and Perceptions of Using Zoom during COVID-19 Pandemic*. International Journal of Technology in Education and Science, 2020. **4**: p. 335-342.

10. Sabzwari, S., 'Rethinking Assessment in Medical Education in the time of COVID-19'. MedEdPublish, 2020. **9**(1): p. 80.

11. David, N., *E-assessment by design: using multiple-choice tests to good effect*. Journal of Further and Higher Education, 2007. **31**(1): p. 53-64.

12. Costello, E., J. Holland, and C. Kirwan, *The future of online testing and assessment: question quality in MOOCs*. International Journal of Educational Technology in Higher Education, 2018. **15**(1): p. 42.

13. Altshuller, G.S., *The innovation algorithm: TRIZ, systematic innovation and technical creativity*. 1999: Technical innovation Center, Inc.

14. Al-Betar, M.A., O.A. Alomari, and S.M. Abu-Romman, *A TRIZ-inspired bat algorithm for gene selection in cancer classification*. (1089-8646 (Electronic)).

15. Swee, N.S.L., et al., *Applying Triz for Production Quality Improvement*. MATEC Web Conf., 2017. **95**.

16. Garber, M.C., S.A. Huston, and C.R. Breese, *Sources of stress in a pharmacy student population*. Curr Pharm Teach Learn, 2019. **11**(1877-1300 (Electronic)): p. 329-337.

17. Vyas, K.S., T.D. Stratton, and N.S. Soares, *Sources of medical student stress*. Educ Health (Abingdon), 2017. **30**(1469-5804 (Electronic)): p. 232-235.

18. Hundertmark, J.A.-O., et al., *Stress and stressors of medical student near-peer tutors during courses: a psychophysiological mixed methods study*. BMC Med Educ, 2019. **19**(1472-6920 (Electronic)): p. 95.

19. Karimi, A. and R. Manteufel. *Use of True-False or Multiple Choice Questions in Measuring and Improving Student Knowledge of Fundamental Concepts in Thermal Science Class*. in *ASEE Gulf-Southwest Section Annual Conference*. 2017. University of Texas, Dallas: The University of Texas at Dallas.