43

D. Nishihama et al. / AR Chemistry: Building up Augmented Reality for Learning Chemical Experiment

AR Chemistry : Building up Augmented Reality
for Learning Chemical Experiment
Daiki Nishihama† , Tatsushi Takeuchi† , Yosuke Horiuchi† , Tomoo Inoue‡ and Ken-ichi Okada††

Graduate School of Science and Technology Keio University, Japan{nishihama, takeuchi, yosuke, okada}@mos.ics.keio.ac.jp‡
Graduate School of Library, Information and Media Studies University of Tsukuba, Japaninoue@slis.tsukuba.ac.jp
Abstract - Learning is changing behavior and ideas by storing experiences and acquiring knowledge. We focus attention on Chemistry in various learning. This paper describes
a learning assist system for chemical experiments ”ARChemistry” which is used in augmented reality and tabletop interface for combining physical papers and digital content.
In this system, tabletop interface shows needed related information on tabletop interface. Augmented reality enables
us to show chemical reactions by using a virtual model. The
user’s history is stored for the feedback of the experiment.
ARChemistry’s feedback function enable us realize mistakesthat are made in the ”Which procedure”, ”Who”, and ”Whatoperation” aspects of the experiment.There are no methods tocatch mistakes on traditional experiment.
Through experimental use, the results showed that an augmented reality interface system worked appropriately.
Keywords: tabletop, augmented reality, collaborative work,
multi user, chemistry

1 Introduction
Learning is to change thinking and actions by experienceand storing knowledge, for example, remembering how to get
to a new office, making a presentation or memorizing new
words.
In this research, we focus on academic learning. There aremany different types of learning. Collaborative learning is
one type, in which, several people work together to achieve
a common goal. Collaborative learning often takes place inchemistry courses during group learning experiments. We decided that this is one effective way of learning[1][2].
Many researchers have studied collaborative learning systems that use a tabletop interface or tangible device. Recently
systems have been developed with large displays that allow
two or more users to intuitively operate the device[3][4][5].
These systems allow users interact with physical objects andretrieve relevant digital information. This paper will propose”ARChemistry” should be used in introductory chemistry courses. The system assists the chemical experiment learning byusing augmented reality (AR) and physical objects. The system has an experiment study space with the tabletop interface,
and we apply that to chemical experiment. When students
use the AR loupe for an experiment they will be able to use
physical objects view 3D animations and retrieve any needed
information like moving ion, chemical reactions. Augmented
Reality offers a way to merge virtual graphics with the realworld in real time[4]. Users will be able to see invisible objects with the system.
By Storing history information the system allows users tosearch for and find their mistakes in the ”Which procedure”,”Who”, and ”What operation” aspects of the experiment. With
the conventional experiment it is impossible to refer to history information to find mistakes. It was impossible to refer
the history information, but our system enables us to get the
feedback. It is important for student to identify and isolate
their mistakes. To confirm the utility of this system, an assessment experiment and user discussion were done.

2 Related Work
Learning is our ordinaries of life and it influences our operation afterward. Everyone has done it regardless of consciousor unconscious consideration. There are many different typesof learning and an Academic learning is a conscious learning.
It falls into two general categories of learning that one isto learn from a textbook and a reference book or that anotheris to learn from the out-of-door activity and experiment.We
focus on the experiment learning of the experience type[5].

2.1 Experiment Learning
We view the experiment learning as a collaborative learning that some facing people work jointly at the same time.
The second learning style is class room learning that teacher
stands in front of students. Learners often achieve high result with the collaborative learning more than with classroom
study[1].
In this case, important points are evaluation to group andpersonal responsibility. Evaluation to group has all learners
in group effects in aiming to achieve high level. Working inthem responsibility, learners have a great awareness and geta high result. Because two factors are direct motivations forlearning. Based on these factors, we focus on the chemical
experiment that many students do it at the same time and thatchemical reaction is very impressive.

ISSN1883-4566/09 © 2010 - Informatics Society and the authors. All rights reserved

International Journal of Informatics Society, VOL. 2, NO.2 (2010) 43-48

2.2 Problem of Experiment Learning
In this research, we make the virtual chemical experiment
to assist chemical learning on the tabletop interface. In viewof the following factors, we designed the system in order todo such the experiment as actual experiment.
• Cooperation with text study• Experiment accident• Experimenter’s error in operation• Experimental environment

44

experiment is finished while not correcting a mistake. However, warning and feedback became possible by recording
each user’s record. Warning is to inform them of the thingthat causing to the accident when keeping experimenting or
the experiment cannot be correctly ended while working on
the tabletop interface. Feedback is to be able to confirm andisolate whether it made a mistake in ”Who”, ”Which procedure”, and ”What operation” because the procedure and theuser’s correspondence are made the text sentences after working, it was not possible to do by an actual experiment.
We aspire achievement of construction that virtual experiment system has four necessary elements.

First, it is difficult for people to always put all studied information in the head while experiment. But students need many
chemical information in experiment to understand invisible
reaction. For example, combining ion and electoron/otherions , features of cation, how to use instrument and so on.
As a result, students often refer to the text or just do without
right information.
Second, chemical accident and user’s mistake will be caused
even if we pay very attention to them.
Third, chemical experiment needs equipment mostly which
is not enough at home.

3 ARChemistry
ARChemistry is a supporting system for chemical experiment study. The system applies 3DCG to virtual experimentspace on tabletop. If the above-mentioned problem is based,
it becomes an important point in the following four points.

Figure 1: Hardware Construction• Visalization by AR• Display related information in experiment• The experimenter’s identification• Record of experiment procedure
At first, movement of ion and electron is ordinarily invisible but it becomes visible by AR. Visualization by AR adds
the understanding more than an actual experiment[8][9]. Repeating is important in study. But the repeating in experimentneeds many kinds of chemicals and needs experimental instruments which we don’t have usually at home. Visualization allows us using chemicals and instruments in a repetitive
manner. For example HCl or H2S in chemicals and the gas
burner etc can be used. These reduce cost and risks. Additionally, this system leverages not only AR but also physicalobjects. It becomes possible not to lose the experiment feeling by operating physical objects like beakers.
Second, if we access an object, system displays some related information on tabletop. Mainly the information displays ionization tendency or contents of beakers and so on.
This operation bridges the gap between text study and experiment study.
Third, identification and record allow users to search forand find mistakes. Correcting work is difficult in an actual
experiment though some chance of study are ruined when the

3.1 Implementation
This system keeps the horizontal and arranges a large-scale
touch sensor display （DiamondTouch[6][7]）as shown in
Figure 1. The electronic information is projected from the
top of the rack, and the physical objects are on the display.
We links the information and physical objects with tabletop
interface. When user operates a beaker , the system displays
beaker’s contents information. In addition, we built up the ARloupe which has the Web camera behind a small display. We
install the AR loupe by each user’s left side. When AR loupe
captured an AR markers, AR loupe will display some 3DCG
about experiment which are images or some 3D animation.
The system progresses order by order in the following threeparts.
• operation physical object• display AR• operation with tabletop interface
It explains details and the manner of operation of these threeparts in order.

45

D. Nishihama et al. / AR Chemistry: Building up Augmented Reality for Learning Chemical Experiment3.1.1 Operation Physical Object
In this system, user can recognize the beakers as a lab ware.
The experiment feeling is not ruined because we move thephysical object on the touch sensor display, and the system
can recognize it.
To recognize the physical object with the system, the copperplate and the copper wire with high conductance are put asshown in Figure 2.

Figure 4: Physical Object Recognition
3.1.2 Display AR
We can use AR loupe for viewing virtual objects. As Figure 5,
AR loupe is constructed by a 9 inch display and Web camera（Qcam Pro for Notebooks）. AR loupes are arranged for all
three user’s left side. Flexible arm fixes them strongly.

Figure 2: Overview of Physical Object With Copperplate
A slight current of electricity from DiamondTouch passes
the user’s body when they operate a physical object, and it isread in the system[6][7]. DiamondTouch is the multi touch
sensor display which can recognize the multi point and who
use the system. Moreover DiamondTouch can distinguish thephysical object according to the size of the sheet copper.
By using this function we can mix contents of the beakers
virtually . As Figure 3, user grips the beaker firmly for eliminating a gap between hand and beakers. When we down the
beaker on the tabletop , it was recognized. If the operation
was correctly, included information would be shown on yourspace at Figure 4. And user select blending button when lasttwo beakers have been recognized to the system. A marker includes text information and 3D animation, appears with soundeffects. User can check the chemical reaction with AR loupe
and text about experiment on tabletop.

Figure 5: Operating AR Loupe
AR loupe distinguishes the presence of the AR marker throughthe image which Web camera captured. When the AR marker
exists, loupe’s display shows 3DCG on the marker after distinguish the shape. Moreover, because the AR marker has
been linked not only 3DCG but also chemical information,
textual information is shown on the tabletop and 3DCG is
possible to display in loupe’s display.
Because the AR loupe can be freely moved by using thearm as shown in Figure 5, it is possible to operate the system
without closing both hands on the touch sensor display.
When users want to see 3DCG, they can pull AR loupe andpeer into the screen of own small display. The display showssome animation. For example flame reaction or a precipitateby uniting ion and so on.
3.1.3 Operation with Tabletop Interface

Figure 3: Gripping Physical Object

We perform procedure chiefly by using the touch sensor display. Figure 6shows an actual operation screen.

International Journal of Informatics Society, VOL. 2, NO.2 (2010) 43-48

Figure 6: An Operation Screen on Tabletop Interface

46

Figure 7: Overview of Apparatus Panel

It is possible to operate the system intuitively with touch
sensor display. Additionally, We innovate a way to use menu
list like a usual PC for handling so operating system is moreeasily. There are some panels to play the system on tabletop
display. In figure 6, there are ”menu list” at upper blue menu
panel and ”AR panel” which has an alphabet on black square.
1. an individual note pad panel
When user operates the system, chemical informationof operated object is shown to each user’s front panel.
The information will be shown in touching AR panel,in gripping a beaker, and after chemical reaction.
2. an indication panel
This panel shows what you must do next. If you mistake the procedure, the system alerts that you are wrong
with warning tone and message.
3. hiding button in menu list
It hides the marker which user feels obstructive at operating it.
4. log button in menu list
When this button is pushed, select panel appears. And
on that panel, user decides whether they log or not, andwhat log file name is.
5. blending button in menu list
As shown in Figure 4, system memorize informationsof two beakers. When the indication panel says thatyou must blend two beakers, this button is possible touse. At blending, AR panel shows 3D animation ofchemical reaction process.
6. apparatus button in menu list
In touching apparatus button, chemical apparatus panel
appears as shown Figure 7. When user touches iconin this panel, system shows paired AR marker whichcorresponds to each operation such as lab wares anda reagent.Figure 8 seemed to select a gas burner icon,
and to have looked into the marker with the AR loupe.
When user touch again ,the gas burner reacts anotherFigure 8: Apparatus：A Gas Burner for Flame Reaction
flame color of the classified ion and user confirms whether
it is a specific ion.
There are filtration, to pour heated water, the outbreak of thehydrogen sulfide as well as a gas burner.

3.2 Record
When we do the experiment operation , our system records
not only the operation but also a physical object and an operator. In a word, Users can confirm ”Which procedure”, ”Who”,
and ”What operation” aspects with the historical information
whether they made a mistake. As a result, in actual experiments it was difficult to specify a mistake of procedure but
our system allow users to discover a critical point in the experiment. And the system allow users to be able to prevent theaccident before happens in real experiment because they aretaught by warning message which shows that you must fail if
you keep to do your operation.

47

D. Nishihama et al. / AR Chemistry: Building up Augmented Reality for Learning Chemical Experiment

4 User Study
4.1 Overview
We conducted a user study to clarify the effects of ARChemistry on users. The objective of this study was to observe how the system would affect the users’ chemical experiment, and how users would interact with the system. Eighteen university participants aged 21 to 24 years (14 men and4 females) were asked to participate in this experiment. The
participants worked in pairs during the experiment. All of theparticipants were inexperienced in regards to chemical activity, and were novice users of the interface that we developed.
We used HMD in order to compare ARloupe to a systemwith HMD. The participants received an explanation of how
to use each system before starting the experiment ? cation
analysis. The participants were asked to accomplish chemical
experiment without a time restriction using each system. The
experiment had the following two system setup: DiamondTouch and ARloupe, and DiamondTouch and HMD.
• ARloupe: Participants had a small display to capture
markers and to show 3DCG. They were able to change
display’s positions freely anytime so they can see both
surface of DiamondTouch and ARloupe separately.
• HMD: Similarly participants used a HMD to display
3DCG. HMDs were eMagin Z800 3D Visor and i-visor
FX601 - SVGA 800 × 600. We placed a web cam to
HMD in order to use it by video see-through type.
We wanted to understand how ARChemistry would allow participants inexperienced in chemical experiment to accomplish
cation analysis. We observed the user interaction in each
setup, and analyzed whether ARChemistry supported user activities related to chemical experience even if the users had
little or no experience in chemical. After accomplishing thecation analysis, we asked participants to fill the questionnaire(ten stage evaluation and free description column).

4.2 Result
Through this experiment, we were able to confirm some
practical advantages of ARChemistry in the process of accomplishing the cation analysis. The Table 1 shows the average time and standard deviation to finish the experiment. Participants with ARloupe finished more fast than with HMD.
The considerable factor of this difference is visibility. User
with HMD said that he had been surprised at suddenly user’s
hand had broken into his sight while he had focused on thetask at hand. And another said that it had been difficult for
him to read text on surface by image degradation through the
HMD. This is why watching at text on surface of DiamondTouch with HMD is smudgy. Users with ARloupe can watch
at the text by naked eyes and check 3DCG in small display sothey can have broad eyeshot.
The Table 2 shows the outcome of a questionnaire. Whenthe summary is seen, the high reputation value is obtained
for the question No.1, No.5, No.8 that we emphasized these

Table 1: Average Time and Standard Deviation to Accomplish The Experiment
MethodARloupe
HMD

Average Time
479.4
751.8

Standard Deviation
102.3
155.2

Table 2: Summary of Questionnaire Response.
No.
1.
2.
3.
4.
5.
6.
7.
8.
9.

Question
I found easy to manipulate table top
I found it intuitive to use beakers
I performed AR Loupe as intended
I imagined the actual chemical
I found it supporting experiment
to use 3D or text information
I feel fun to use this system
I found Chemistry of interest
I found easy to understandchemical reaction
I found the system available forreview and preparation

Average
8.0
7.2
7.4
7.5
8.1
8.7
6.5
8.1
7.7

themes when we made the system. It says that the participants were able to use the system easily by touching the tabletop screen or using physical objects (beakers) or ARloupe.
Responses say that ARChemistry’s operation was easy because it was a very simple operation. About No.5, No.8, they
say that 3DCG improves users understanding. In comparison
with text learning, virtual chemical reactions are so clear thatusers see how some ions react.
In addition, the question No.6, No.9 was the high reputation values too. These comments are that this experiment wasso fun because we had been speaking each other while operating and that it was possible to operate the system without getting tired. Participants seemed to actively interact with ARChemistry while communicating with each other and to provide understanding of chemical experiment. Efficiency goesup to study enjoying it, so this evaluation backs up the utilityof this system that these evaluations.
On the other hand, No 7 is lower score than other question.
This is why participants who were not interest in chemicalfrom the beginning didn’t get their interest from the systema short time later even if that operating is fun and easy tounderstand.

5 Conclusion
There are styles of the learning now for many divergences.
Interaction methods with AR and a touch sensor display increased slowly in that. The system ”ARChemistry” used AR
and tabletop interface and put a focus for the chemical experiment learning. In this system, tabletop interface can show
needed related chemical information on tabletop and how you
push forward operation. Augmented reality allow us to showchemical reaction by using a virtual model. And the feedbackof the experiment is enabled by storing user’s history. The

International Journal of Informatics Society, VOL. 2, NO.2 (2010) 43-48

feedback is able to realize us the mistakes made ”Which procedure”, ”Who”, and ”What operation” aspects.They are new
method to combine text study and experiment one in chemical. Through evaluation experiment, the results showed thatthe system of an augmented reality interface worked appropriately and is interesting in learning and discussion each other.

48

Daiki Nishihama received a B.S. degree from the
Department of Information and Computer Scienceof Keio University, Japan in 2009. He is currently
working toward an M.S. degree in Open and Environment Systems at Keio University. His researchinterests include face-to-face collaborative work.
He received the Student award from IWIN’09.

Acknowledgements
We would like to acknowledge Mitsubishi Electric ResearchLaboratories for donating DiamondTouch table.

REFERENCES[1] Masaya Okada, Akimichi Yamada, Mizuki Yoshida, Hiroyuki Tarumi, Takanobu Kayugawa, Kazuyuki Moriya: Collaborative Environmental Learning with the DigitalEE II System Augmenting Real and Virtual Experiences IPSJ Journal,vol45-1,pp 229-243, 2004.
[2] Masaya Okada, Hiroyuki Tarumi, Tetsuhiko Yoshimura,Kazuyuki Moriya, Tetsuro Sakai : DigitalEE: A Support
System for Collaborative Environmental Education Using Distributed Virtual Space,Systems and Computers inJapan, vol.33, No.8,pp 51-63, 2002.
[3] Diana Africano and Sara Berg, Kent Lindbergh and Peter Lundholm and Fredrik Nilbrink and Anna Persson: Designing tangible interfaces for children’s collaboration, CHI ’04 extended abstracts on Human factors incomputing systems, pp.853-868 ,2004.
[4] Kikuo Asai and Tomotsugu Kondo1 : Molecular Structure Visualization System Using Augmented Reality
Journal of Multimedia Aided Education Research 2008,
Vol. 4, No. 2, 37 − 43
[5] H.Koike ，Y.Sato and Y.Kobayashi: Integrating paperand digital information on EnhacedDesk: a method forreal-time finger tracking on augmented desk system,
ACM Trans. on Computer-Human Interaction, vol.8,
No.4,pp.307-322,2001.
[6] Keigo Kitahara,Tomoo Inoue,Hiroshi Shigeno,Ken-ichi
Okada: A Tabletop Interface for Supporting Collaborative Learning, IPSJ Journal, Vol.47, No.11, pp.30543062, 2006.
[7] Dietz, P., Leigh, D., ”DiamondTouch: A Multi-User
Touch Technology”, ACM Symposium on User Interface Software and Technology (UIST), pp.219-226,
2001.
[8] Moffat Mathews, Madan Challa, Cheng-Tse Chu, Gu
Jian, Hartmut Seichter, Raphael Grasset : Evaluation of
spatial abilities through tabletop AR CHINZ ’07: Proceedings of the 7th ACM SIGCHI New Zealand chapter’s international conference on Computer-human interaction: design centered HCI ,vol.254, pp 17-24, 2007.
[9] Fotis Liarokapis, Robert M. Newman:Design experiences of multimodal mixed reality interfaces Proceedings of the 25th annual ACM international conferenceon Design of communication,pp 34-41 , 2007.
(Received August 30, 2009)(Revised June 19, 2010)

Tatsushi Takeuchi received a B.S. degree fromthe Department of Information and Computer Science of Keio University, Japan in 2008. He is currently working toward an M.S. degree in Open andEnvironment Systems at Keio University in 2010.
He is currently working at TV Asahi Corporation.

Yosuke Horiuchi received a B.S. degree from the
Department of Information and Computer Scienceof Keio University, Japan in 2010. He is currently
working toward an M.S. degree in Open and Environment Systems at Keio University. His researchinterests include face-to-face collaborative work.

Tomoo Inoue is an Associate Professor in the Graduate School of Library, Information and Media
Studies at University of Tsukuba His research interests include understanding and supporting collaboration on the information networks and in thephysical world, and its application to education.He
received his B.S., M.S., and Ph.D. degrees from
Keio University in 1992, 1994 and 1998 respectively. He visited the University of California, Irvine
as a visiting researcher from 1996 to 1997. Healso worked at Tokyo Denki University, National
Center for Science Information Systems, and National Institute of Informatics in Japan.

Kenichi Okada received B.S., M.E and ph.D degrees in instrumentation engineering from Keio
University, in 1973, 1975 and 1982, respectively.
He is currently a professor at the Department ofInformation and Computer Science at Keio University. His research interests include CSCW, groupware, human-computer interaction and mobile computing. He is a member of IEEE, ACM, and IPSJ.
He was a chairman of SIGGW, achief editor of
the IPSJ Journal, and an editor of IEICE Transactions.Dr.Okada received the IPSJ Best Paper Awardin 1995, 2001, 2008, as well as the IPSJ 40th Anniversary Paper Award in2000.