Quiz Generation on the Electronic Guide Application for
Improving Learning Experience in the Museum
Masaki Ueta

Tomoya Hashiguchi

Huu-Long Pham

aa20t502@ai.u-hyogo.ac.jp
University of Hyogo
Hyogo, Japanaa19j508@ai.u-hyogo.ac.jp
University of Hyogo
Hyogo, Japanaa19e510@ai.u-hyogo.ac.jp
University of Hyogo
Hyogo, Japan

Yoshiyuki Shoji

Noriko Kando

Yusuke Yamamoto

shoji@it.aoyama.ac.jp
Aoyama Gakuin University
Kanagawa, Japankando@nii.ac.jp
National Institute of Informatics
Tokyo, Japan

yamamoto@inf.shizuoka.ac.jp
Shizuoka University
Shizuoka, Japan

Takehiro Yamamoto

Hiroaki Ohshimat.yamamoto@sis.u-hyogo.ac.jp
University of Hyogo
Hyogo, Japanohshima@ai.u-hyogo.ac.jp
University of Hyogo
Hyogo, Japan

ABSTRACT
We propose a method to generate quizzes on a museum electronic
guide application. While a museum is considered to be a place forlearning, it would be hard for a visitor to actively appreciate theexhibits in the museum, especially when they have little knowledge about the exhibits. In this study, we develop a method that
automatically generates quizzes about the exhibits on an electronic
guide application. The proposed method utilizes a BERT modelthat is trained with the additional corpus constructed from the descriptions of the exhibits and automatically generates a quiz about
exhibit. By solving quizzes about the exhibits during the museum
visit, we expect that a visitor’s museum experience would be more
active, and they would understand the exhibits more deeply. We
implement the proposed method on the electronic guide applicationthat is designed for the National Museum of Ethnology, Japan (a.k.a.
Minpaku).

KEYWORDS

Figure 1: An example quiz on the Minpaku’s electronic guideapplication (originally in Japanese)

Museum visit, Quiz generation, Personalization

1

INTRODUCTION

There are many museums around the world. A museum is a place
to learn about culture and history through exhibits. However, domuseum visitors learn anything while they visit? If so, do museum
visitors remember what they learned in the museums?
Many people who visit the museum simply vaguely look at theexhibits, or lose their direction due to a large number of exhibits andthe amount of information related to them. Such a visit will not beremembered and will not be established as knowledge. Therefore,
in this study, we used quizzes about exhibits to enhance the learning experiences of museum visitors. By solving quizzes about theexhibits during the museum visit, a visitor can think more actively
and understand the exhibits more deeply.
Let us think of an example. Now, a museum visitor is looking at
gongs used in Cambodian rituals. The visitor uses the electronic
guide application to take a quiz about the gongs. The visitor needs

to actively refer to the actual exhibits and their descriptions tolook for hints of the quiz’s answer. By taking quiz on Cambodian
gongs, the visitor may also develop an interest in the “rituals” andthe “musical instruments”. This will lead to the appreciation ofsimilar exhibits, such as the vertical flute used in Bulgarian rituals.
Therefore, the visitor may also try to answer a quiz on the Bulgarianvertical flute. Based on the knowledge gained during the visit, thevisitor can compare the exhibits and discover similarities between
them, which can lead to a deeper understanding of the exhibits.
This way of viewing the exhibits is considered to be a great learning
experience for the visitor.
In this study, we focus on the National Museum of Ethnology,
Japan (a.k.a. Minpaku). We implement the proposed method on the
Minpaku’s existing electronic guide application, which we havedeveloped before. A visitor can use the Minpaku’s electronic guideapplication and enjoy quizzes during the visit. An example quiz isshown in Figure 1.

2

RELATED WORKIn recent years, research on the use of digital device in museums is
conducted. Klopfer et al. [3] conducted a riddle solving game using
digital device for museum education. The purpose of this study is
to encourage visitors to refer to the exhibits and communication
among visitors. As a result, visitors were actively looking for andreferring to the exhibits that provided the answers. It was alsosuggested that the problem of visitors focusing only on digital
devices could be solved by incorporating a riddle solving game.
Robert et al. [6] showed that by using quizzes in museums, visitors
actively referred to both the exhibit descriptions and the electronic
guides.
In a museum visit, it is important to provide contents according to
the visitor’s interests. There have been studies on personalization of
museum contents. Wang et al. [8] [9] thought that the information
should be based on the visitor’s own interests and context. Basedon this idea, they developed a personalized museum visit programusing the visitor’s interest and context information. Dijk et al. [7]used game-style questions about the topic of the exhibit at the
beginning of the museum visit. Using the answers to the questions,they showed visitors a personalized visit route. Kuflik et al. [5]obtained information about visitors’ needs and interests. Using
this information, they developed a graph-based recommendation
system that recommends relevant information from the museum’s
own information.

3

Figure 2: Visitors admiring the exhibits• Used to send [ ? ], such as during rituals(1) signal(2) message(3) letterOf the choices, “signal” is the correct choice.

3.2

In this study, we use descriptions of the exhibits to generate quizzes.
Descriptions are extracted from the database provided by the Minpaku, which contains detailed information and images about theexhibits in the Minpaku. The database contains 72,428 items of data.
Table 1 shows an example of data in the Minpaku’s database. The
descriptions about an exhibit contains the following information:• Usage,• Fabrication method and materials,• Transition and distribution,• Other information.
We use the above information for quiz generation.

QUIZ GENERATION METHODThis section describes the quiz generation method. First, we describe
problem definition in 3.1 and the data was used in this study in 3.2.
Approach to quiz generation and details of the system are describedin 3.3 and 3.4, respectively.

3.1

Database of the National Museum of
Ethnology

Problem Definition

In this study, we develop a system for generating quizzes. A quiz
to be generated is a three-choice question. There quiz question is asentence which is missing a word. The goal is to guess the correct
word from three choices.
First, we explain the problem definition for the quiz generation
system. The input and the output of the quiz generation system is
as follows.

3.3

Approach to Quiz Generation

In this study, we use a language model called BERT [1] to generate
quizzes. Typically, a language model is used to predict the next word
in a sentence using only previous words. BERT is able to predict aword in the middle of a sentence given words from both sides. For
example, given the sentence “The [ ? ] of Japan is Tokyo”, BERT canfill the word “capital” into the blank position, using information of
the rest words. Output of BERT is not only the best word, but also
other candidates and their probabilities. We use the same approachto generate incorrect choices by using predictions of BERT on aquiz question that has a blank part.
We use the pre-trained BERT model for Japanese published by
Inui and Suzuki Laboratory at Tohoku University. The model was
pre-trained with data from Japanese Wikipedia. In this study, we
perform additional training on the above pre-trained BERT modelusing the descriptions of the exhibits from the database provided
by Minpaku.

Input A sentence.
Output The given sentence which has one word removed andthree choices to be filled in the removed part. One word is
the correct choice (the removed word itself). The rest twowords are the incorrect choices.
We will give a concrete example using an actual exhibit in the
Minpaku. Suppose that a visitor is looking at a drum made by carving wood, which is used in the rituals of a country in the Oceania
region (Figure 2). From the description of the exhibit, it contains theinformation “Used to send signal, such as during rituals”. This description has the information “ritual”. The generated quiz questionand choices are as follows:2

Table 1: An example of data in the Minpaku’s database
Exhibit ID
Exhibit name
User
Location
Collection date
Usage and how to use
Fabrication method and materials

K0006979
Splintered wood drumSpecific adult men
New Hebrides islands, Ambrym island, Ranon village
1969-01-14
Music instrument. Used to send signal, such as during rituals.
Beat the body with a wooden stick.
Fabrication method:Carving
Material:wood

In addition, we also use Japanese WordNet [2] for quiz choices
generation. WordNet is a thesaurus database systematized by hypernym or hyponym. We use WordNet as a filter to remove inappropriate candidate words outputted by BERT.

3.4

them as incorrect choices of the quiz. In the phenomenon thatsome of the candidates are hypernyms or hyponyms sof the correct
choice, the quiz becomes inappropriate. To solve this problem, we
use Japanese WordNet to find out hypernyms and hyponyms of thecorrect choice from candidates outputted by BERT. The incorrect
choices of a quiz are selected so that there are no hypernyms andhyponyms of the correct choice.
We explain details of the process in this step using an example.
Consider that quiz question and correct choice generated from thefirst step is as follows:
Quiz question: Used to send [ ? ], such as during rituals.
Correct choice: signal.
We use BERT to predict the “[ ? ]” part of the quiz question. The
output candidates are:
Candidates: gift, food, sign, sentence, letter, thing, article, cue
Since the correct choice is signal, we use WordNet to find out hypernyms and hyponyms of signal from the above candidates, which
are: sign and cue, and remove them from the list of candidates. After
that, we randomly select two words from the remaining candidates
and use them as quiz’s incorrect choices, which can be:
First incorrect choice: message
Second incorrect choice: letter

Quiz Generation System

This section describes the quiz generation system. The quiz generation system consists of the two steps: quiz question generationand incorrect choices selection The process of the quiz generation
system is described below with an example using the exhibit shownin Figure 2.
3.4.1 Quiz question generation. The first step is to generate thequiz question. Quiz question is generated from the description ofa exhibit by replacing one word from it with the notation “[ ? ]”.
The word to be replaced is a noun, since nouns usually provide
important information of a sentence. To select the replaced word,
we firstly use the morphological analysis software MeCab [4] tosplit the description into parts of speech, get all the nouns andrandomly select one word. Some nouns such as pronouns are set
as stop words since the may not directly represent the content,
and will not be selected. The quiz question is then generated byreplacing the selected word in the description with the notation “[? ]”. The “[ ? ]” in the description indicates that the part is a blank.
For example, consider the case where the description is “Used tosend signal, such as during rituals”. The nouns in the description
are: signal and ritual. We randomly select one word from these two.
If “signal” is selected, the quiz question will be as follows:
Quiz question: Used to send [ ? ], such as during rituals.
The selected word, which is signal in this case, is the correct choiceamong the quiz choices.

4

USER EXPERIMENT AND DISCUSSION

This sections describes the user experiment conducted in this research and its results.

4.1

Experimental Setup

A user experiment has been conducted to evaluate the proposed
quiz generation system. We prepared two group of subjects where
the first group consists of seven university students and the second group consists of four university students. Subjects from bothgroups use the Minpaku’s electronic guide application during an
actual two-hour visit to Minpaku. Each group took the visit withthe following conditions:Group 1: Use the electronic guide application without quizsystem.
Group 2: Use the electronic guide application with quiz system.
After the visit, subjects from both groups were asked whether theexperience of a visit to museum was improved or not using theelectronic guide application. In this study, we used a questionnaire

3.4.2 Incorrect choices selection. The second step is to select quiz’s
incorrect choices. This step generates incorrect choices that properly fit into the blank in a quiz question. The BERT language modelis used in this step. In this study, we use a BERT model which has
been pre-trained using from Japanese Wikipedia. We also implemented additional training on the pre-trained model using descriptions of exhibits from Minpaku’s database.
Since BERT is able to give predictions of a blank part in a sentence using the rest words, we used BERT to output candidates
for incorrect choices. Given a quiz question generated from theprevious step which has a word replace by the notation “[ ?]”, we
use BERT to output candidates to be filled in the “[ ? ]” and use
3

Table 2: Questionnaires after observing the user experiment
Question No.
1
2
3
4
5
6
7
8
9
10
11
12
13

questionnaire
Do you like visiting museums.
Did you have a certain idea of what you wanted to see?
Do you have memorable exhibits from past museum visits?
Was the Minpaku’s guide application helpful?
Was the Minpaku’s guide application easy to use?
Were you able to develop an interest in the exhibits?
Could you understand the information about the exhibits?
Did your willingness to look at exhibits increase?
Do you feel that you have gained knowledge about the exhibits?
Did you satisfy with the visit in this time.
How fun was the visit.
Were you able to focus on viewing the exhibition.
Do you want to come to the Minpaku again?

3.57
2.00
2.00
3.14
2.29
3.57
2.71
3.86
3.00
4.14
4.43
3.43
3.86

3.75
3.00
3.25
4.00
3.25
4.75
4.00
4.50
3.50
4.00
4.25
4.00
3.50

This work was supported in part by JSPS KAKENHI Grant Numbers
JP18H03494, JP18H03243, JP18K18161, JP17H00762, and JP16H01756.
It was also supported by the NII Open Collaborative Research 2020
Program, numbers 20S1001, 20AD03 and 20AD04.
We sincerely thank the Japanese National Museum of Ethnology(Minpaku) for offering the exhibit meta-data.

Experimental Results and Discussion

Table 2 shows the average responses to the questionnaire of eachgroup, with quiz system or without quiz system.
Among the questions in the questionnaire, we focus on question
6, 7, 8 and 9. Basically, the average values of the responses for thevisit using the application with the quizzes is higher than without
the quizzes.
In particular, for the seventh question, which is “Could you understand the information about the exhibits?", the average answer
was 2.71 for a visit without the quizzes compared to a value of 4.00
for a visit with the quizzes, which is a significant improvement.
This indicates that the use of quizzes in the visit can deepen the understanding of the exhibits. In addition, the quizzes can be provide
an opportunity for visitors to become interested in the exhibits. In
this way, it was shown that the quiz was effective in improving thelearning experience of the museum visit.

5

With quiz system

ACKNOWLEDGEMENT

with 13 questions, where answers for each question is rated on a
5-point Likert scale where 1=not at all and 5=totally agree.

4.2

Without system

REFERENCES[1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert:
Pre-training of Deep Bidirectional Transformers for Language Understanding.
Journal of arXiv preprint arXiv:1810.04805 (2018).
[2] Hitoshi Isahara, Francis Bond, Kiyotaka Uchimoto, Masao Utiyama, and Kyoko
Kanzaki. 2008. Development of the Japanese WordNet. In Proceedings of the 6thInternational Conference on Language Resources and Evaluation. 2420–2423.
[3] Eric Klopfer, Judy Perry, Kurt Squire, Ming-Fong Jan, and Constance Steinkuehler.
2005. Mystery at The Museum A Collaborative Game for Museum Education. In
Proceedings of the 7th International Conference on Computer Support for Collaborative Learning. 316–320.
[4] Taku Kudo, Kaoru Yamamoto, and Yuji Matsumoto. 2004. Applying conditional random fields to Japanese morphological analysis. In Proceedings of the 9th Conferenceon Empirical Methods in Natural Language Processing. 230–237.
[5] Tsvi Kuflik, Einat Minkov, and Keren Kahanov. 2014. Graph-based Recommendation in the Museum.. In Proceedings of the 1st International Workshop on Decision
Making and Recommender Systems. 46–48.
[6] Jessica Roberts, Amartya Banerjee, Annette Hong, Steven McGee, Michael Horn,and Matt Matcuk. 2018. Digital Exhibit Labels in Museums: Promoting Visitor
Engagement with Cultural Artifacts. In Proceedings of the 36th CHI Conference on
Human Factors in Computing Systems. 1–12.
[7] Elisabeth Van Dijk, Andreas Lingnau, and Hub Kockelkorn. 2012. Measuring
Enjoyment of An Interactive Museum Experience. In Proceedings of the 14th ACM
International Conference on Multimodal Interaction. 249–256.
[8] Yiwen Wang, Natalia Stash, Lora Aroyo, Peter Gorgels, Lloyd Rutledge, and Guus
Schreiber. 2008. Recommendations Based on Semantically Enriched Museum
Collections. Journal of Web Semantics 6, 4 (2008), 283–290.
[9] Yiwen Wang, Natalia Stash, Rody Sambeek, Yuri Schuurmans, Lora Aroyo, Guus
Schreiber, and Peter Gorgels. 2009. Cultivating Personalized Museum Tours Online
and On-site. Journal of Interdisciplinary Science Reviews 34, 2–3 (2009), 139–153.

CONCLUSION

In this study, we proposed a method to enhance the learning experience of visitors to museums using quizzes about exhibits.
We developed a system that automatically generates quizzes fromdescriptions of exhibits using a fine-tuned BERT model. The system
was implemented on the Minpaku’s electronic guide application.
By solving quizzes about the exhibits during the museum visit, avisitor can think more actively and understand the exhibits moredeeply.
We also conducted an user experiment to evaluate the developed
system. Participants of the experiment used the Minpaku’s electronic guide application with the quiz system installed during an
actual visit to Minpaku, and answered to several questions at theend of the visit. As a result, we confirm that resolving quizzes during a museum tour is effective in getting interested in the exhibits
and understanding the information about the exhibits.
4